Piano fingering generation algorithm combining curved structure and local adjacency information

Through the piano fingering generation algorithm combining curved structure and local adjacency information, the dot product self-attention mechanism and the truncated residual module are used to solve the problems of insufficient utilization ability of curved structures and error propagation in the existing technology, and high precision is achieved The piano fingering generation expands the application scope of automatic marking of music score fingering.

CN118917358APending Publication Date: 2024-11-08TIANJIN UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202310505520.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-05-08
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

The existing piano fingering generation algorithm cannot effectively utilize the curved structure and local adjacency information, resulting in error propagation problems in the generated fingering and cannot generate high-quality fingering that can be played.

Method used

Using a piano fingering generation algorithm combining curved structure and local adjacency information, the music score features are extracted and the adjacency of the current time step is combined with the structure extractor based on the dot product self-attention mechanism. Information, use the Pytorch deep learning framework for feature learning and parameter training to generate high-quality fingering that can be played.

Benefits of technology

It improves the accuracy of fingering generation, solves the problems of insufficient utilization ability of curved structures and error propagation, and provides a feasible solution for automatic marking of fingering of music scores.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118917358A_ABST
    Figure CN118917358A_ABST
Patent Text Reader

Abstract

The invention discloses a piano fingering generation algorithm combining a curved structure and local adjacency information, relates to the technical field of deep learning, is used for solving the problem of automatic generation of piano music score fingering, and comprises the following steps: step 1, preprocessing original music score data, denoising the original music score by using a normalization method, and obtaining a denoising result; converting a data form by using a one-hot coding technology; 2, constructing a fingering generation model combining global structure information and local adjacency information by using a Pytorch framework, wherein the main body structure of the model is divided into three parts: a structure extractor model used for extracting music score style structure features, a truncated residual network used for emphasizing adjacency music score information in a decoding stage, and a fingering generator used for generating fingering by autoregression; and 3, testing the model, storing optimal model parameters, and generating a fingering result by using the method.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular to a piano fingering generation algorithm combining musical form structure with local adjacency information. Background Art

[0002] Piano fingering is an important factor that affects the effect of piano performance, and it is also a learning problem encountered by piano beginners. However, a large number of scores lack fingering annotations, which brings great trouble to the performers. For scores without fingering annotations, determining their fingering requires repeated attempts based on experience, which is extremely difficult for piano beginners. Even for experienced pianists, the process of determining the fingering of new scores is not easy. Using computing technology to automatically annotate fingerings for scores can broaden the range of music scores for beginners and remove the primary obstacle to performance; it can also provide a reference for experienced performers to explore appropriate fingerings, thereby effectively promoting the development of piano teaching, expanding the number of piano players, and protecting the interest in piano practice.

[0003] With the development of deep learning technology, LSTM models with good time series modeling capabilities have achieved performance that exceeds traditional statistical learning models in various time series processing tasks. Due to LSTM's excellent sequence processing performance, LSTM models have become the mainstream model in fingering generation tasks in recent years. However, the piano fingering generation algorithm based on LSTM has the problem of not being able to utilize the form structure and generating errors.

[0004] Compared with the fingering generation method based on LSTM, the fingering generation method based on dot product self-attention can better extract the global structural information of the music. However, like the LSTM model, when only using the encoder-decoder architecture network Transformer based on the dot product self-attention module, the decoder does not explicitly use the adjacent music score information of the current time step when generating fingerings, which to a certain extent causes the model to have error propagation problems. The error propagation problem will make the fingering quality generated by the model unplayable. Directly using the dot product self-attention-based method cannot obtain usable and reasonable fingering results. Summary of the invention

[0005] In view of the problem that the current fingering generation model is insufficient in utilizing the form structure of the score and the generation model has error propagation, the present invention proposes a piano fingering generation algorithm that combines the form structure with local adjacency information. The technical solution of the present invention is as follows:

[0006] 1. Comprehensively extract the pitch features and speed features in the music score, including the following steps:

[0007] The first step is to preprocess the score information. The score information in the PIG dataset and the self-built dataset is serialized to extract the pitch sequence and the time series of the notes. At the same time, the normalization technology is used to extract the real time value and note density of the time series, and the one-hot encoding technology is used to extract the one-hot vector of the pitch sequence.

[0008] The second step is to use the Pytorch deep learning framework to build the model of the present invention and perform feature learning and parameter training. The model is divided into the following three parts:

[0009] (1) Structure Extractor Based on Dot Product Self-Attention Mechanism

[0010] The structure extractor based on the dot product self-attention mechanism follows the encoder structure of the Transformer. The dot product self-attention module is used to extract the repetitive form information of the score, the position encoding feedforward network is introduced to improve the model's ability to fit the source data, and the ELU activation function is used to avoid the dead neuron problem caused by the ReLU function in the original Transformer model.

[0011] (2) Truncated residual module

[0012] The truncated residual module truncates the source music information, uses residual calculation and normalization technology, combines the adjacent score information of the current time step with the generated partial fingering information, and participates in the subsequent fingering judgment.

[0013] (3) Autoregressive Fingering Generator

[0014] The autoregressive fingering generator also uses the dot product self-attention mechanism to calculate the self-attention value of the music sequence, uses the masked self-attention technique to simulate the autoregressive characteristics of the fingering sequence, and introduces the Softmax function and greedy decoder at the output to generate fingerings.

[0015] The present invention applies deep learning technology to the field of piano score fingering generation. Aiming at the problem that the existing fingering generation algorithm has insufficient form structure utilization ability and the generation model has error propagation, a piano fingering generation algorithm combining form structure and local adjacency information is proposed to generate high-quality fingering that can be played. The present invention improves the accuracy of fingering generation and provides the possibility of being widely used in the field of automatic fingering annotation of music scores. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0017] Figure 1 A structural diagram of a piano fingering generation algorithm combining musical form structure and local adjacency information provided by an embodiment of the present invention;

[0018] Figure 2 A flow chart of a method for extracting comprehensive features of music scores provided by an embodiment of the present invention;

[0019] Figure 3 A structural diagram of a structure extractor provided by an embodiment of the present invention;

[0020] Figure 4 A structural diagram of a position encoding feedforward network provided by an embodiment of the present invention;

[0021] Figure 5 A structural diagram of a truncated residual module provided in an embodiment of the present invention;

[0022] Figure 6 A structural diagram of a fingering generator provided by an embodiment of the present invention; DETAILED DESCRIPTION

[0023] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work belong to the scope of protection of the present invention.

[0024] Figure 1 The structure diagram of the piano fingering generation algorithm combining the musical form structure and the local adjacency information provided by the embodiment of the present invention is shown, and its main modules include music score information preprocessing, structure extractor, truncated residual module and fingering generator.

[0025] The first step is preprocessing of music score information.

[0026] Figure 2 The flowchart of the music score comprehensive feature extraction method provided by the embodiment of the present invention is shown. The music score information in the PIG dataset and the self-built dataset is serialized to extract the pitch sequence and the time series of the notes. At the same time, the normalization technology is used to extract the real time value and the note density of the time series, and the one-hot encoding technology is used to extract the one-hot vector of the pitch sequence.

[0027] Music score feature preprocessing uses the pitch sequence p of the original music and the start time t of each note on and end time t off , comprehensive data preprocessing is performed on the original music score.

[0028] The pitch information reflects the position of the fingers when playing. Based on the original pitch sequence p, the author extracts the unique hot vector of the pitch The pitch differential encoding d, black key identifier and chord identifier are used as pitch-related features. Among them, the pitch unique hot vector That is the one-hot encoding of pitch MIDI.

[0029] The basic idea of ​​pitch differential coding d is to use the difference between adjacent pitches to represent the relative distance between keys. The calculation method is:

[0030]

[0031] Where: d(t) is the generated pitch difference code of the current time step, x represents the original pitch MIDI, t is the time step variable, k represents the number of notes contained in the chord, and if it is a single tone, k is set to 0.

[0032] The black key identifier or chord identifier is a Boolean value, which, when set to 1, indicates that the key corresponding to the current pitch is a black key or the current pitch is part of a chord. The reason for considering black keys or chords is that some specific fingerings cannot be used when the fingers are pressed on black keys or on chords that are pressed together.

[0033] The definition of density is the normalized value of the sum of all notes that will sound within one second after the current note starts. Density represents the frequency of fingers pressing keys. The author normalizes the density of each note in the dataset, and the calculation method is:

[0034]

[0035] Where: density′ is the starting time t of the note on The total number of notes that sound within one second after the sound is played, MEAN(density′) is the average value of all density′ in the data set, and STD(density′) is the standard deviation of all density′ in the data set.

[0036] The true duration is defined as the note ending time t off and start time t on The real time value represents the time that the finger stays on the key. It is calculated as:

[0037] duration = t on -t off

[0038] The second step is to use the Pytorch deep learning framework to build the model of the present invention, and perform feature learning and parameter training. The model consists of three parts: structure extractor, truncated residual module and fingering generator:

[0039] (1) Structure Extractor Based on Dot Product Self-Attention Mechanism

[0040] Figure 3 FIG. 1 shows a structure diagram of a structure extractor provided by an embodiment of the present invention. The structure extractor follows the encoder structure of the Transformer, such as Figure 3 First, the model uses the product self-attention module to extract the repetitive form information of the score, and its calculation process is:

[0041]

[0042] Among them, d k is a scaling factor used to stabilize training; Q, K, and V are the query vector, key vector, and value vector, respectively, and all three vectors are obtained from each element in the input sequence through different linear transformations.

[0043] Multiple dot product self-attention modules form multi-head attention, and the calculation process is:

[0044] Multihead(Q,K,V)=concat(head1,head2,...,head i )W out

[0045] head i =Attention(Q i W i Q , K i W i K , V i W i V )

[0046] Then, the position encoding feedforward network is introduced to improve the model's ability to fit the source data. Its structure is as follows Figure 4 As shown. First, the encoding process of position encoding can be expressed as:

[0047]

[0048] E t ′=E t +P t

[0049] Secondly, residual connection and layer normalization technology are introduced to deepen the number of training layers and improve the model's expressiveness. The process can be expressed as follows:

[0050] Y = x + SubBlock (LayerNorm (x))

[0051] The second half of the structure extractor uses a positional encoding feed-forward neural network layer to add the model's ability to learn nonlinear transformations, such as Figure 4 As shown, the process is expressed as:

[0052] E=g(WX+B)

[0053] In the formula, are the trainable model weights, is the input matrix, is the trainable bias of the fully connected layer. is the output of the fully connected layer. g() is the activation function.

[0054] Because when the ReLU activation function is used, if the input has a negative value, the "dead neuron" phenomenon may occur, which will lead to instability in model training. The present invention chooses to introduce the ELU activation function instead of the ReLU function commonly used in the source Transformer model. The calculation process of the ELU activation function is:

[0055]

[0056] Where x is the eigenvalue of the input ELU function.

[0057] The ELU function does not output 0, but a negative saturated function value. In addition, the output of the ELU function is smooth, while the output of the ReLU function is piecewise linear. This means that the ELU function is smoother than the ReLU function when calculating the gradient, so it is easier to optimize the model. In addition, when the input is positive, the ELU function has similar performance to the ReLU function, and when the input is negative, the ELU function has better performance than the ReLU function. This can improve the prediction accuracy of the neural network.

[0058] To sum up, the calculation process of the position encoding feedforward network is expressed as:

[0059]

[0060] Where: x is the input feature value, w1 and w2 are the trainable weights of the two fully connected layers, and b1 and b2 are the trainable biases of the two fully connected layers.

[0061] (2) Truncated residual module

[0062] The truncated residual module truncates the source music information, uses residual calculation and normalization technology, combines the adjacent music score information of the current time step with the generated partial fingering information, and participates in the subsequent fingering judgment. Its structure is as follows Figure 5 As shown, the calculation process of this module can be expressed as:

[0063]

[0064] Where: is the output of the truncated residual module, L t is the cutoff length, which is the length of the currently decoded fingering plus 1, indicating that the model will intercept the music score information of the next step. is the output of the score embedding module, i.e., the score feature sequence from which features have been extracted. O is the previous fingering output, which represents the generated fingering. The shift() function shifts the fingering output right by 1 and adds a special custom identifier at the beginning to indicate the start of the sequence. truncate() means to replace I e Press L t Length truncation.

[0065] (3) Autoregressive Fingering Generator

[0066] The autoregressive fingering generator also uses the dot product self-attention mechanism to calculate the self-attention value of the music sequence, uses the masked self-attention technique to simulate the autoregressive characteristics of the fingering sequence, and introduces the Softmax function and greedy decoder at the output to generate fingering. Its basic structure is similar to that of the structure extractor, such as Figure 6 As shown in the figure. The difference from the structure extraction is that the fingering generator introduces a masked multi-head attention mechanism (Masked Multi-head Attention) module. The calculation process of the masked multi-head attention mechanism can be expressed as:

[0067] Mask(X)=MeX

[0068]

[0069] in is the input sequence of the masked self-attention module. The mask matrix is a lower triangular matrix with the same dimensions as the input matrix X, and all the left sides of its diagonal are 1 and all the right sides are 0. e stands for element-by-element product, which means that after element-by-element product of the input matrix X, the attention matrix will retain the value before the current time step, masking the subsequent fingering information. In this way, fingerings can be generated time-by-time during generation, thus simulating the actual situation of human judgment.

[0070] The calculation method of the multi-head self-attention module of the fingering generator is the same as that of the multi-head self-attention module in the global feature extractor. The difference is the input information. The multi-head self-attention module here accepts the global self-attention output from the global structure extractor as the input source of its K matrix and V matrix, while the Q matrix comes from the previous submodule.

[0071] In addition, the fingering generator receives the attention value containing the form structure information output by the structure extractor during the second multi-head self-attention mechanism. At the end of the fingering generator, the Softmax function is used to select the fingering with the largest probability of occurrence.

[0072] The third step is to test the network and save the best network model parameters.

[0073] The PIG dataset and the proprietary dataset were used as test sets for this experiment, and the model was trained using the 7-fold cross-validation method.

[0074] (1) Load the optimal network parameters obtained from feature training into the network and split the test set;

[0075] (2) The obtained segmentation result image was compared with the original data for performance evaluation, and the average fingering matching rate was 66.56% and the peak matching rate was 73.53%.

[0076] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. The piano fingering generation algorithm combining the musical form structure and local adjacency information includes the following steps: The first step is to preprocess the score information. The score information in the PIG dataset and the self-built dataset is serialized into notes, and the pitch sequence and the time series of the notes are extracted. At the same time, the real time value and the density of the notes of the time series are extracted, and the one-hot encoding technology is used to extract the one-hot vector of the pitch sequence. The second step is to use the Pytorch deep learning framework to build the model of the present invention and perform feature learning and parameter training. The model is divided into the following three parts: (1) Structure Extractor Based on Dot Product Self-Attention Mechanism The structure extractor based on the dot product self-attention mechanism follows the structure of the Transformer encoder. The dot product self-attention module is used to extract the repetitive form information of the score, the position encoding feedforward network is introduced to improve the model's ability to fit the source data, and the ELU activation function is used to avoid the dead neuron problem caused by the ReLU function in the source model. (2) Truncated residual module The truncated residual module truncates the source music information, uses residual calculation and normalization technology, combines the adjacent score information of the current time step with the generated partial fingering information, and participates in the subsequent fingering judgment. (3) Autoregressive Fingering Generator The autoregressive fingering generator also uses the dot product self-attention mechanism to calculate the self-attention value of the music sequence, and uses the masked self-attention technique to simulate the autoregressive characteristics of the fingering sequence. In the third step, the model is tested and the best model parameters are saved. The Soffmax function is used to obtain the fingering distribution probability, and the greedy decoder is used to generate the final fingering result.