A method, device, and electronic device for marking fingering in piano music scores.

Abstract

This application discloses a method, apparatus, and electronic device for marking fingering in piano sheet music, relating to the field of piano technology, and invented to improve the effect of fingering marking. The method includes: determining the position of the key corresponding to each marking unit on the piano and whether the corresponding key is a black or white key; determining the distance between the key corresponding to each marking unit in the sheet music and the key corresponding to the previous marking unit; obtaining a first feature vector based on the pitch of each marking unit, the position of the corresponding key on the piano, whether the corresponding key is a black or white key, and a first network layer in a preset fingering generation model; obtaining a second feature vector based on the first feature vector, the distance between the key corresponding to each marking unit and the key corresponding to the previous marking unit, and a second network layer in the preset fingering generation model; and using the second feature vector as input to a conditional random field in the preset fingering generation model to obtain the fingering sequence for each marking unit.

Description

Technical Field

[0001] This application relates to the field of piano technology, and in particular to a method, device, electronic device, and readable storage medium for marking fingering in piano music scores. Background Technology

[0002] Proper fingering can increase the fluency of playing; theoretically, every piano score has an optimal fingering sequence. Automatic fingering annotation in piano score can eliminate early playing obstacles for amateur players and provide initial reference fingering for professional pianists. Applied to music education, automatic fingering annotation can generate optimal fingering for beginners, preventing them from practicing with incorrect fingering. The method of automatic fingering annotation can also be extended to more general robotic tasks using humanoid robotic hands.

[0003] Although the finger annotation problem is similar to the time series annotation problem in natural language processing, see [link to relevant documentation]. Figure 1 However, it differs from typical time series labeling problems. Typical time series labeling problems have only one input and one output at any given moment, while fingering labeling problems have one input and one output at any given moment for single notes, and chords can have multiple inputs and multiple outputs at the same time.

[0004] In existing technology, chords are simply expanded into a sequence according to pitch from high to low or from low to high and input as multiple single notes into the same network layer to obtain the corresponding fingering. See [link to relevant documentation]. Figure 2 However, this method fails to consider adjacent chords and single notes, as well as the relationship between chords, resulting in poor fingering notation. Summary of the Invention

[0005] In view of this, embodiments of this application provide a method, apparatus, electronic device, and readable storage medium for marking fingerings in piano sheet music, thereby improving the effect of fingering marking.

[0006] In a first aspect, embodiments of this application provide a method for marking fingering in piano sheet music, comprising: determining the pitch of a marking unit in the sheet music to be marked; wherein the marking unit includes a single note or a chord; determining, based on the pitch of each marking unit, the position of the corresponding key on the piano and whether the corresponding key is a black key or a white key; determining, based on the position of the corresponding key on the piano, the distance between the key corresponding to each marking unit and the key corresponding to the previous marking unit in the sheet music to be marked; and determining, based on the pitch of each marking unit and the position of the corresponding key on the piano, the distance between the key corresponding to each marking unit and the key corresponding to the previous marking unit; and determining, based on the pitch of each marking unit and the position of the corresponding key on the piano, the distance between the key corresponding to each marking unit and the key corresponding to the previous marking unit; and determining, based on the pitch of each marking unit and the position of the corresponding key on the piano, the distance between the key corresponding to each marking unit and the key corresponding to the previous marking unit. The first feature vector is obtained by considering the position, whether the key corresponding to each labeled unit is a black or white key, and the first network layer in the preset fingering generation model. A second feature vector is obtained based on the first feature vector, the distance between the key corresponding to each labeled unit and the key corresponding to the previous labeled unit, and the second network layer in the preset fingering generation model. The second feature vector is used as the input to the conditional random field in the preset fingering generation model to obtain the fingering sequence for each labeled unit. The obtained fingering sequences for each labeled unit are then used to label each labeled unit in the score to be labeled.

[0007] According to a specific implementation of the embodiments of this application, the step of determining the position of the piano key corresponding to each marked unit and whether the piano key corresponding to each marked unit is a black key or a white key based on the pitch of each marked unit and a preset correspondence between the pitch and the position of the piano key in the piano includes: determining the position of the piano key corresponding to each marked unit and whether the piano key corresponding to each marked unit is a black key or a white key based on the pitch of each marked unit and the preset correspondence between the pitch and the position of the piano key in the piano.

[0008] According to a specific implementation of an embodiment of this application, the step of obtaining a first feature vector based on the pitch of each labeled unit, the position of the key corresponding to each labeled unit on the piano, whether the key corresponding to each labeled unit is a black key or a white key, and the first network layer in the preset fingering generation model includes: determining the interval of each labeled unit; and obtaining the first feature vector based on the pitch of each labeled unit, the position of the key corresponding to each labeled unit on the piano, whether the key corresponding to each labeled unit is a black key or a white key, the interval of each labeled unit, and the first network layer in the preset fingering generation model.

[0009] According to a specific implementation of an embodiment of this application, the annotation unit is a single note; wherein, determining the interval of each annotation unit includes: determining the interval of the single note as a preset value; and / or, the annotation unit is a chord, the chord including at least two notes; wherein, determining the interval of each annotation unit includes: sorting the pitches of each note of the chord in descending or ascending order; the sorted pitches include a first pitch and a second pitch adjacent to and following the first pitch; subtracting the second pitch from the first pitch to obtain the difference between the two, and determining the difference as the interval of the note corresponding to the second pitch.

[0010] According to a specific implementation of an embodiment of this application, the preset fingering generation model further includes an embedding layer; the embedding layer is connected to the first network layer and the second network layer respectively; obtaining the first feature vector based on the pitch of each labeled unit, the position of the key corresponding to each labeled unit on the piano, whether the key corresponding to each labeled unit is a black key or a white key, the interval of each labeled unit, and the first network layer in the preset fingering generation model includes: encoding the pitch of each labeled unit, the interval of each labeled unit, the position of the key corresponding to each labeled unit on the piano, and whether the key corresponding to each labeled unit is a black key or a white key according to a preset rule, and encoding the pitch, the interval of each labeled unit, the position of the key corresponding to each labeled unit on the piano, and the key corresponding to each labeled unit according to a preset rule. The black or white keys are used as input to the embedding layer to obtain a first encoding vector corresponding to each labeled unit; each of the first encoding vectors is input to the first network layer of a preset fingering generation model to obtain a first feature vector; the step of obtaining a second feature vector based on the first feature vector, the distance between the key corresponding to each labeled unit and the key corresponding to the previous labeled unit, and the second network layer of the preset fingering generation model includes: encoding the distance between the key corresponding to each labeled unit and the key corresponding to the previous labeled unit according to a preset rule, and using the encoded distance as input to the embedding layer to obtain a second encoding vector corresponding to each distance; using the first encoding vector and the second encoding vector as input to the second network layer to obtain the second feature vector.

[0011] According to a specific implementation of this application, the sheet music to be annotated includes a first annotation unit and a second annotation unit; the first annotation unit and the second annotation unit are adjacent; the first annotation unit is a single note, and the second annotation unit is a single note; determining the distance between the key corresponding to each annotation unit and the key corresponding to the previous annotation unit in the sheet music to be annotated, based on the position of the key corresponding to each annotation unit on the piano, includes: determining a first position of the key corresponding to the first annotation unit on the piano, and a second position of the key corresponding to the second annotation unit on the piano; determining the distance between the key corresponding to the first annotation unit and the key corresponding to the second annotation unit based on the first position and the second position; or, the first annotation unit is a chord, and the second... The unit of annotation is a single note; determining the distance between the key corresponding to each annotation unit and the key corresponding to the previous annotation unit in the score to be annotated, based on the position of the key corresponding to each annotation unit on the piano, includes: determining the positions of the keys corresponding to each note in the first annotation unit on the piano, and the third position of the key corresponding to the second annotation unit on the piano; determining the positions of the keys corresponding to each note in the first annotation unit on the piano, and the third position of the key corresponding to the second annotation unit on the piano; determining the fourth position of the key corresponding to the first annotation unit on the piano based on the positions; determining the distance between the key corresponding to the first annotation unit and the key corresponding to the second annotation unit based on the third and fourth positions; or,

[0012] The first annotation unit is a chord, and the second annotation unit is a chord; determining the distance between the key corresponding to each annotation unit in the score to be annotated and the key corresponding to the previous annotation unit, based on the position of the key corresponding to each annotation unit on the piano, includes:

[0013] Determine the positions of the keys corresponding to each note in the first annotation unit on the piano, and the positions of the keys corresponding to each note in the second annotation unit on the piano.

[0014] Based on the positions corresponding to the first marked unit, determine the fifth position of the piano key corresponding to the first marked unit in the piano;

[0015] Based on the positions corresponding to the second annotation unit, determine the sixth position of the key on the piano corresponding to the second annotation unit;

[0016] Based on the fifth position and the sixth position, the distance between the piano key corresponding to the first annotation unit and the piano key corresponding to the second annotation unit is determined.

[0017] According to one specific implementation of this application, the first network layer is a Long Short-Term Memory (LSTM) network; the second network layer is a bidirectional LSM network; or, the first network layer is a fully self-attention network and the second network layer is a bidirectional LSM network; or, the first network layer is an LSM network and the second network layer is a fully self-attention network; or, the first network layer is a fully attention network and the second network layer is also a fully attention network.

[0018] Secondly, embodiments of this application provide a piano sheet music fingering annotation device, comprising: a first determining module, configured to determine the pitch of annotated units in the sheet music to be annotated; wherein the annotated units include single notes or chords; a second determining module, configured to determine, based on the pitch of each annotated unit, the position of the corresponding key on the piano and whether the corresponding key is a black key or a white key; a third determining module, configured to determine, based on the position of the corresponding key on the piano, the distance between the key corresponding to each annotated unit and the key corresponding to the previous annotated unit in the sheet music to be annotated; and a first obtaining module, configured to determine, based on the pitch of each annotated unit and the position of the corresponding key on the piano, the distance between the key corresponding to each annotated unit and the key corresponding to the previous annotated unit .... The system uses a first feature vector to obtain the following: the position of the keys on the piano, whether the key corresponding to each marked unit is a black or white key, and the first network layer in a preset fingering generation model; a second feature vector to obtain the second feature vector based on the first feature vector, the distance between the key corresponding to each marked unit and the key corresponding to the previous marked unit, and the second network layer in the preset fingering generation model; a third feature vector to use the second feature vector as input to a conditional random field in the preset fingering generation model to obtain the fingering sequence for each marked unit; and a marking module to use the obtained fingering sequence for each marked unit to mark each marked unit in the score to be marked.

[0019] According to a specific implementation of an embodiment of this application, the second determining module is specifically used to: determine the position of the key corresponding to each marked unit in the piano and whether the key corresponding to each marked unit is a black key or a white key, based on the pitch of each marked unit and the preset correspondence between the pitch and the position of the key in the piano.

[0020] According to a specific implementation of an embodiment of this application, the first obtaining module includes: a determining submodule, used to determine the interval of each marked unit; and an obtaining submodule, used to obtain the first feature vector based on the pitch of each marked unit, the position of the key corresponding to each marked unit in the piano, whether the key corresponding to each marked unit is a black key or a white key, the interval of each marked unit, and the first network layer in the preset fingering generation model.

[0021] According to a specific implementation of an embodiment of this application, the annotation unit is a single note; wherein, the determining submodule is specifically used to: determine the interval of a single note as a preset value; and / or, the annotation unit is a chord, the chord including at least two notes; wherein, the determining submodule is specifically used to: sort the pitch of each note of the chord in descending or ascending order; the sorted pitches include a first pitch and a second pitch adjacent to and following the first pitch; subtract the second pitch from the first pitch to obtain the difference between the two, and determine the difference as the interval of the note corresponding to the second pitch.

[0022] According to a specific implementation of an embodiment of this application, the preset fingering generation model further includes an embedding layer; the embedding layer is connected to the first network layer and the second network layer respectively; the obtaining submodule is specifically used for: encoding the pitch of each annotation unit, the interval of each annotation unit, the position of the key corresponding to each annotation unit on the piano, and whether the key corresponding to each annotation unit is a black key or a white key according to a preset rule, and using the encoded pitch, the interval of each annotation unit, the position of the key corresponding to each annotation unit on the piano, and whether the key corresponding to each annotation unit is a black key or a white key as input to the embedding layer to obtain a first encoding vector corresponding to each annotation unit; inputting each first encoding vector into the first network layer of the preset fingering generation model to obtain a first feature vector; the second obtaining module is specifically used for: encoding the distance between the key corresponding to each annotation unit and the key corresponding to the previous annotation unit according to a preset rule, and using the encoded distance as input to the embedding layer to obtain a second encoding vector corresponding to each distance; using the first encoding vector and the second encoding vector as input to the second network layer to obtain a second feature vector.

[0023] According to a specific implementation of an embodiment of this application, the sheet music to be annotated includes a first annotation unit and a second annotation unit; the first annotation unit and the second annotation unit are adjacent; the first annotation unit is a single note, and the second annotation unit is a single note; the third determining module is specifically used to: determine the first position of the key corresponding to the first annotation unit on the piano, and the second position of the key corresponding to the second annotation unit on the piano; determine the distance between the key corresponding to the first annotation unit and the key corresponding to the second annotation unit based on the first position and the second position; or, the first annotation unit is a chord, and the second annotation unit is a single note; the third determining module is specifically used to: determine the positions of the keys corresponding to each note in the first annotation unit on the piano, and the third position of the keys corresponding to the second annotation unit on the piano; determine the fourth position of the key corresponding to the first annotation unit on the piano based on the positions; and determine the distance between the keys corresponding to the first annotation unit and the keys corresponding to the second annotation unit on the piano based on the positions. The third and fourth positions determine the distance between the keys corresponding to the first and second annotation units; or, the first and second annotation units are both chords; determining the distance between the keys corresponding to each annotation unit and the keys corresponding to the previous annotation unit in the score to be annotated, based on the position of the keys corresponding to each annotation unit on the piano, includes: determining the positions of the keys corresponding to each note in the first annotation unit and the positions of the keys corresponding to each note in the second annotation unit on the piano; determining the fifth position of the keys corresponding to the first annotation unit on the piano based on the positions corresponding to the first annotation unit; determining the sixth position of the keys corresponding to the second annotation unit on the piano based on the positions corresponding to the second annotation unit; and determining the distance between the keys corresponding to the first and second annotation units based on the fifth and sixth positions.

[0024] According to a specific implementation of an embodiment of this application, the first network layer is a long short-term memory network; the second network layer is a bidirectional long short-term memory network; or, the first network layer is a fully self-attention network and the second network layer is a bidirectional long short-term memory network; or, the first network layer is a long short-term memory network and the second network layer is a fully self-attention network; or, the first network layer is a full attention network and the second network layer is also a full attention network.

[0025] Thirdly, embodiments of this application provide an electronic device, which includes: a housing, a processor, a memory, a circuit board, and a power supply circuit, wherein the circuit board is disposed inside the space enclosed by the housing, and the processor and the memory are disposed on the circuit board; the power supply circuit is used to supply power to various circuits or devices of the above-mentioned electronic device; the memory is used to store executable program code; the processor runs a program corresponding to the executable program code by reading the executable program code stored in the memory, for executing the piano sheet music fingering notation method described in any of the foregoing implementations.

[0026] Fourthly, embodiments of this application provide a computer-readable storage medium storing one or more programs, which can be executed by one or more processors to implement the piano sheet music fingering annotation method described in any of the foregoing implementations.

[0027] The piano sheet music fingering annotation method, device, electronic device, and readable storage medium of this embodiment obtain a first feature vector based on the pitch of each annotation unit, the position of the corresponding key on the piano, whether the corresponding key is a black or white key, and the first network layer in a preset fingering generation model. Then, a second feature vector is obtained based on the first feature vector, the distance between the key corresponding to each annotation unit and the key corresponding to the previous annotation unit, and the second network layer in the preset fingering generation model. The second feature vector is used as input to the conditional random field in the preset fingering generation model to obtain the fingering sequence for each annotation unit. This obtained fingering sequence is then used to annotate each annotation unit in the sheet music to be annotated. Because the pitch of each annotation unit, the position of the corresponding key on the piano, and the first network layer in the preset fingering generation model are all considered, a first feature vector is obtained. The second feature vector is then used as input to the conditional random field in the preset fingering generation model to obtain the fingering sequence for each annotation unit. Finally, the fingering sequence for each annotation unit is used to annotate each annotation unit in the sheet music to be annotated. The black or white key corresponds to its own annotation unit, that is, the annotation unit at the same moment corresponds to the musical score characteristics and piano key characteristics. Combined with the first network layer, the first feature is determined. In addition, the distance between the piano key corresponding to each annotation unit and the piano key corresponding to the previous annotation unit represents the characteristics corresponding to the annotation unit at different times. Combined with the second network layer, the second feature is determined. Therefore, in this embodiment, the pitch corresponding to its own annotation unit, the position of the piano key corresponding to each annotation unit in the piano, whether the piano key corresponding to each annotation unit is a black or white key, and the characteristics corresponding to the annotation unit at different times, that is, the distance between the piano keys, are input into different network layers for processing respectively, so that the fingering sequence of each annotation unit is more accurate. Then, the obtained fingering sequence is used to annotate each annotation unit in the musical score to be annotated, thereby improving the fingering annotation effect. Attached Figure Description

[0028] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0029] Figure 1 This refers to the time series labeling model in existing natural language processing technologies;

[0030] Figure 2 A schematic diagram of a single-layer sequence for labeling piano fingering in existing technology;

[0031] Figure 3 A flowchart illustrating a piano score fingering notation method provided in an embodiment of this application;

[0032] Figure 4 This is a schematic diagram of the structure of a specific fingering generation model in this application;

[0033] Figure 5 This is a schematic diagram of a piano fingering annotation model including two-layer sequences in one embodiment of this application;

[0034] Figure 6 This is a schematic diagram of the structure of a piano sheet music fingering annotation device provided in an embodiment of this application;

[0035] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0036] The embodiments of this application will now be described in detail with reference to the accompanying drawings. It should be understood that the described embodiments are merely some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0037] To enable those skilled in the art to better understand the technical concept, implementation scheme and beneficial effects of the embodiments of this application, detailed descriptions are provided below through specific embodiments.

[0038] One embodiment of this application provides a method for marking fingering in piano sheet music, which improves the effect of fingering marking.

[0039] Figure 3 This is a flowchart illustrating a piano sheet music fingering notation method provided in one embodiment of this application, as shown below. Figure 3 As shown, the piano sheet music fingering notation method of this embodiment may include:

[0040] S101. Determine the pitch of the units marked in the musical score to be annotated.

[0041] In this embodiment, the labeling unit includes a single note or a chord. A single note corresponds to one musical note, and a chord corresponds to multiple musical notes.

[0042] S102. Based on the pitch of each marked unit, determine the position of the key corresponding to each marked unit on the piano and whether the key corresponding to each marked unit is a black key or a white key.

[0043] Each marked unit corresponds to a piano key. Therefore, based on the pitch of each marked unit, we can determine the position of the key corresponding to each marked unit and whether the key corresponding to each marked unit is a black key or a white key.

[0044] In some embodiments, determining the position of the key corresponding to each marked unit on the piano and whether the key corresponding to each marked unit is a black key or a white key based on the pitch of each marked unit (S102) may include:

[0045] S102a. Based on the pitch of each marked unit and the preset correspondence between the pitch and the position of the key in the piano, determine the position of the key corresponding to each marked unit in the piano and whether the key corresponding to each marked unit is a black key or a white key.

[0046] Each key on a piano corresponds to a pitch. Therefore, based on the pitch of each marked unit and the preset correspondence between the pitch and the position of the key on the piano, the position of the key corresponding to each marked unit on the piano can be determined. Furthermore, while determining the position of the key on the piano, it is possible to determine whether the key is a black key or a white key.

[0047] S103. Based on the position of the key corresponding to each annotation unit on the piano, determine the distance between the key corresponding to each annotation unit in the score to be annotated and the key corresponding to the previous annotation unit.

[0048] For a musical score to be annotated that includes a first annotation unit and a second annotation unit; where the first annotation unit is adjacent to the second annotation unit, in some cases, both the first and second annotation units are single notes; determining the distance between the key corresponding to each annotation unit and the key corresponding to the previous annotation unit in the musical score to be annotated (S103) based on the position of the key corresponding to each annotation unit on the piano may include:

[0049] S103a. Determine the first position of the key in the piano corresponding to the first annotation unit, and the second position of the key in the piano corresponding to the second annotation unit.

[0050] S103b. Based on the first position and the second position, determine the distance between the piano key corresponding to the first annotation unit and the piano key corresponding to the second annotation unit.

[0051] The difference between the first position and the second position is the distance between the piano keys corresponding to the first annotation unit and the piano keys corresponding to the second annotation unit.

[0052] It is understandable that in the sheet music to be annotated, the first annotation unit is placed after the second annotation unit.

[0053] In other examples, the first annotation unit is a chord, and the second annotation unit is a single note; in this embodiment, determining the distance between the key corresponding to each annotation unit and the key corresponding to the previous annotation unit in the score to be annotated (S103) based on the position of the key corresponding to each annotation unit on the piano may include:

[0054] S103c, Determine the positions of the keys corresponding to each note in the first annotation unit on the piano, and the third position of the key corresponding to the second annotation unit on the piano.

[0055] S103d. Based on each position, determine the fourth position of the key corresponding to the first marked unit in the piano.

[0056] In some embodiments, such as when a chord includes an odd number of notes, such as three, and the three notes correspond to three keys, the position of the key in the middle position on the piano can be determined as the position of each note on the piano, that is, the three notes correspond to the same position on the piano.

[0057] In some other cases, regardless of whether the chord includes an odd or even number of notes, the average position of all notes on the piano can be taken as the position of the key on the piano corresponding to each note.

[0058] S103e. Based on the third position and the fourth position, determine the distance between the piano key corresponding to the first annotation unit and the piano key corresponding to the second annotation unit.

[0059] The difference between the third and fourth positions is the distance between the keys corresponding to the first and second annotation units.

[0060] Understandably, in the sheet music to be annotated, if the first annotation unit is after the second annotation unit, S103e obtains the distance between the key corresponding to the first annotation unit and the key corresponding to the second annotation unit; if the second annotation unit is after the first annotation unit, S103e obtains the distance between the key corresponding to the second annotation unit and the key corresponding to the first annotation unit.

[0061] In some other examples, the first annotation unit is a chord, and the second annotation unit is a chord; the step of determining the distance between the key corresponding to each annotation unit in the score to be annotated and the key corresponding to the previous annotation unit based on the position of the key on the piano (S103) may include:

[0062] S103f: Determine the positions of the keys corresponding to each note in the first annotation unit on the piano, and the positions of the keys corresponding to each note in the second annotation unit on the piano.

[0063] S103g: Based on the positions corresponding to the first annotation unit, determine the fifth position of the key in the piano corresponding to the first annotation unit.

[0064] S103h: Based on the positions corresponding to the second annotation unit, determine the sixth position of the key in the piano corresponding to the second annotation unit.

[0065] S103j. Based on the fifth position and the sixth position, determine the distance between the piano key corresponding to the first annotation unit and the piano key corresponding to the second annotation unit.

[0066] The methods for determining each of the above steps can be found in the relevant processes described in the foregoing embodiments.

[0067] If a musical score has a first annotation unit, and the second annotation unit is the one preceding and adjacent to the first annotation unit, and the first annotation unit corresponds to the 5th key from the left on the piano, while the second annotation unit corresponds to the 10th key from the left on the piano, then the distance between the keys corresponding to the first and second annotation units can be determined based on the 5th and 10th keys. Specifically, the distance corresponding to the first annotation unit is determined by subtracting the key position corresponding to the second annotation unit from the key position corresponding to the first annotation unit.

[0068] For cases where the first annotation unit is a single note and the second annotation unit is a chord, the method for determining the position of the chord is the same as the method in the above embodiment.

[0069] S104. Based on the pitch of each labeled unit, the position of the key corresponding to each labeled unit in the piano, whether the key corresponding to each labeled unit is a black key or a white key, and the first network layer in the preset fingering generation model, obtain the first feature vector.

[0070] In some examples, the first network layer can be a Long Short-Term Memory (LSTM) network. LSTM has long-term memory capabilities and can automatically adjust the memory length, discard redundant information, and use information from strongly correlated moments for prediction.

[0071] In this embodiment, the pitch of each annotation unit, the position of the corresponding key on the piano, and whether the corresponding key is a black or white key are all related to their own annotation units. That is, the annotation units at the same moment correspond to the musical score characteristics and the piano key characteristics. Therefore, the first network layer can also be called the same moment sequence layer.

[0072] S105. Based on the first feature vector, the distance between the key corresponding to each labeled unit and the key corresponding to the previous labeled unit, and the second network layer in the preset fingering generation model, a second feature vector is obtained.

[0073] In some examples, the second network layer can be a bidirectional long short-term memory network (Bi-LSTM). Building upon LSTM, Bi-LSTM can also simultaneously take into account context information in both the forward and backward directions.

[0074] The distance between the key corresponding to each annotation unit and the key corresponding to the previous annotation unit represents the distance between adjacent annotation units. That is, it can represent the distance between adjacent single notes and their corresponding keys in the score to be annotated, the distance between adjacent single notes and their corresponding keys to chords, and the distance between adjacent chords and their corresponding keys to chords.

[0075] The distance between the key corresponding to each labeled unit and the key corresponding to the previous labeled unit represents the characteristics of the labeled unit at different times. Therefore, the second network layer can also be called the sequence layer at different times.

[0076] In some examples, the first network layer is a fully self-attention network (Transformer), and the second network layer is a bidirectional long short-term memory network.

[0077] In other examples, the first network layer is a long short-term memory network and the second network layer is a fully self-attention network (Transformer); or both the first and second network layers are fully self-attention networks.

[0078] The Transformer network architecture, built using the attention mechanism, is a transformation model that relies entirely on self-attention to compute the representations of its inputs and outputs. It can better capture long-distance dependencies in sequences, thereby improving the model's performance.

[0079] S106. Use the second feature vector as the input of the conditional random field in the preset finger generation model to obtain the finger sequence for each labeled unit.

[0080] Conditional Random Field (CRF) can obtain the global optimal solution by using the output of the second network layer at each time step as the state feature function of the CRF, that is, the finger sequence of each labeled unit is the global optimal solution.

[0081] S107. Using the fingering sequence of each marked unit obtained, mark each marked unit in the score to be marked.

[0082] In this embodiment, the pitch of each annotation unit in the score to be annotated is determined. Based on the pitch of each annotation unit, the position and key characteristics of the corresponding key on the piano are determined. Based on the position of the corresponding key on the piano, the distance between the key corresponding to each annotation unit and the key corresponding to the previous annotation unit is determined. A first feature vector is obtained based on the pitch of each annotation unit, the position of the corresponding key on the piano, whether the key is black or white, and the first network layer of the preset fingering generation model. A second feature vector is then obtained based on the first feature vector, the distance between the key corresponding to each annotation unit and the key corresponding to the previous annotation unit, and the second network layer of the preset fingering generation model. The second feature vector is used as the input to the conditional random field in the preset fingering generation model to obtain the fingering sequence for each annotation unit. The obtained fingering sequence for each annotation unit is then used to... Each unit in the score to be annotated is annotated. Since the pitch of each unit, the corresponding key position, and whether it is a black or white key are all associated with the unit itself, i.e., the characteristics of the score and the piano keys at the same time, are determined by combining the first network layer to form the first feature. In addition, the distance between the key corresponding to each unit and the key corresponding to the previous unit represents the characteristics of the unit at different times, and is determined by combining the second network layer to form the second feature. Therefore, in this embodiment, the pitch of the unit itself, the position of the key corresponding to each unit on the piano, whether the key corresponding to each unit is a black or white key, and the characteristics of the unit at different times, i.e., the distance between the keys, are input into different network layers for processing. This makes the fingering sequence of each unit more accurate, and then the obtained fingering sequence is used to annotate each unit in the score to be annotated, thereby improving the fingering annotation effect.

[0083] Understandably, the same model can be used to train the annotation units in the sheet music for the left and right hands separately, only the training data for the model will be different. Furthermore, the corresponding fingerings can be obtained for the annotation units of the left and right hands in the sheet music.

[0084] In some cases, for longer musical scores, the score can be segmented, usually at the rest positions.

[0085] To improve the accuracy of the fingering sequence for each labeled unit and further enhance the labeling effect, this application provides another embodiment, which is basically the same as the above embodiment, except that, in this embodiment, the first feature vector (S104) is obtained based on the pitch of each labeled unit, the position of the key corresponding to each labeled unit on the piano, whether the key corresponding to each labeled unit is a black key or a white key, and the first network layer in the preset fingering generation model. This may include:

[0086] S104a, Determine the interval for each marked unit.

[0087] The interval of each marked unit can be determined according to preset rules.

[0088] In some examples, the unit of measurement is a single tone.

[0089] In this embodiment, determining the interval (S104a) of each marked unit may include:

[0090] A1. Set the interval of a single note to the preset value.

[0091] In this embodiment, when the unit of annotation is a single note, the interval of that single note can be determined to be a preset value.

[0092] In some other examples, the unit of measurement is chord, which consists of at least two notes.

[0093] In this embodiment, determining the interval (S104a) of each marked unit may include:

[0094] B1. Sort the pitch of each note of the chord in descending or ascending order; the sorted pitches include the first pitch and the second pitch that is adjacent to and follows the first pitch.

[0095] The pitches of each note in the chord are sorted to obtain the first pitch and the second pitch. The first pitch and the second pitch are adjacent, and the second pitch is located after the first pitch in the sorting.

[0096] B2. Subtract the first pitch from the second pitch to obtain the difference between the two, and determine the interval of the note corresponding to the second pitch.

[0097] When the unit of measurement is a chord, the interval of a note in a chord can be determined by subtracting the pitch of the note from the pitch of the preceding note in the sequence, and then using the difference as the interval of the note.

[0098] It is understandable that the interval of the first note in the sorted notes can be a preset value.

[0099] S104b: Based on the pitch of each labeled unit, the position of the key corresponding to each labeled unit in the piano, whether the key corresponding to each labeled unit is a black key or a white key, the interval of each labeled unit, and the first network layer in the preset fingering generation model, the first feature vector is obtained.

[0100] In this embodiment, based on the pitch of each annotation unit, the position of the key corresponding to each annotation unit in the piano, whether the key corresponding to each annotation unit is a black key or a white key, and the first network layer in the preset fingering generation model, the interval of each annotation unit is added to determine the first feature vector, which can further improve the accuracy of the final fingering sequence and improve the annotation effect.

[0101] Another embodiment of this application is basically the same as the above embodiment, except that the preset finger generation model in this embodiment may further include an embedding layer; the embedding layer is connected to the first network layer and the second network layer respectively.

[0102] Embedding layers can convert the encoded input parameters into vectors.

[0103] In this embodiment, based on the pitch of each labeled unit, the position of the corresponding key on the piano, whether the corresponding key is a black or white key, the interval of each labeled unit, and the first network layer in the preset fingering generation model, a first feature vector (S104b) is obtained, which may include:

[0104] C1. Encode the pitch, interval, position of the corresponding key on the piano, and whether the corresponding key is a black or white key for each labeled unit according to a preset rule. Then, use the encoded pitch, interval, position, and whether the corresponding key is a black or white key as input to the embedding layer to obtain the first encoding vector for each labeled unit.

[0105] In this embodiment, the pitch of each labeled unit, the interval of each labeled unit, the position of the corresponding key on the piano, and the key itself are coded as either a black or white key. Specifically, the pitch code can be determined based on the correspondence between pitch and preset codes; the interval code can be determined based on the correspondence between interval and preset codes; the position code can be determined based on the correspondence between position and preset codes; and the black or white key code can be determined based on the correspondence between the position of the corresponding key on the piano and preset codes, and whether the key is a black or white key. It is understood that different preset codes can be used when determining the codes for pitch, interval, and key features.

[0106] Through the above process, we can obtain the pitch of each labeled unit, the interval of each labeled unit, the position of the key corresponding to each labeled unit on the piano, and whether the key corresponding to each labeled unit is a black key or a white key. The corresponding code is then input into the embedding layer as a whole for each labeled unit to obtain the encoding vector corresponding to each labeled unit, which is the first encoding vector.

[0107] C2. Input each first encoding vector into the first network layer of the preset finger generation model to obtain the first feature vector.

[0108] In this embodiment, the second feature vector (S105) is obtained based on the first feature vector, the distance between the key corresponding to each labeled unit and the key corresponding to the previous labeled unit, and the second network layer in the preset fingering generation model. This may include:

[0109] S105a. Encode the distance between the piano key corresponding to each labeled unit and the piano key corresponding to the previous labeled unit according to a preset rule, and use the encoded distance as the input of the embedding layer to obtain the second encoded vector corresponding to each distance.

[0110] Encode the distance between the key corresponding to each labeled unit and the key corresponding to the previous labeled unit. Specifically, the distance code between the key corresponding to each labeled unit and the key corresponding to the previous labeled unit can be determined based on the correspondence between the distance and the preset code.

[0111] It is understood that the preset encoding in this embodiment is different from the preset encoding in the above embodiments.

[0112] S105b: Using the first encoding vector and the second encoding vector as input to the second network layer, a second feature vector is obtained.

[0113] The following detailed description of the solution in this application is based on a specific embodiment.

[0114] The piano notation method in this embodiment may include:

[0115] Pitch: 88 pitch codes;

[0116] Key positions (index): 52 white keys are coded;

[0117] Key markings: 1 represents a white key, 2 represents a black key;

[0118] The interval at the same moment, i.e., the interval between the current note and the previous note in the chord at the same moment, is determined by sorting the notes in order from high to low or from low to high; it is a fixed value when it is a single note.

[0119] Distance of hand movement at different times: The distance of hand movement between the current time and the previous time, including the distance of hand movement between single notes, between chords, and between chords.

[0120] 1. Model Structure:

[0121] See Figure 4 The overall structure of the model consists of an Embedding layer, a pitch sequence layer at the same time (LSTM), a sequence layer at different times (Bi-LSTM), and a CRF layer.

[0122] 2. The input uses tuple encoding, x i =(seq i ,dist i ), where seq i For sequences at the same time, dist i seq represents the distance the hand moved at different times. i Represented as ..., tuple i ,…,tuple i = (pitch, index, key, interval), where pitch is the pitch, index is the key position, key is the black / white key marker, and interval is the interval distance at any given moment. For a single note, seq i Includes a tuple i For chords, including multiple tuples i , and tuple i The number of tuples is equal to the number of notes in a chord. i These represent the four parameters mentioned above corresponding to a musical note.

[0123] tuple i Each tuple item is used to obtain its embedding vector by querying its embedding matrix. Then, the embedding vectors of each tuple item are concatenated to obtain a tuple. i The embedding vector. (The seq) i The tuple embedding vector sequence is input into the pitch sequence layer (one-way LSTM network) at the same time step, and the output hidden state vector seq is output. ih That is, seq i Feature representation, seq ih and dist i Concatenation is used as the input to sequence layers (bidirectional LSTM network) at different time steps, outputting the hidden state of the bidirectional LSTM at each time step, which is then used as the finger state feature input for CRF. The objective function of the model is to minimize the negative logarithm label sequence score.

[0124] To make the generated finger placement more accurate, in some cases a fully connected network is connected between the bidirectional LSTM and the CRF.

[0125] The training model has 31 possible label combinations: 5 single-finger combinations ("1", "2", "3", "4", "5"), 10 two-finger combinations ("1,2", "1,3", "1,4", "1,5", "2,3", "2,4", "2,5", "3,4", "3,5", "4,5"), and 10 three-finger combinations ("1,2,3", "1,2,4"). The following are some of the four-finger combination codes: "1,2,5", "1,3,4", "1,3,5", "1,4,5", "2,3,4", "2,3,5", "2,4,5", "3,4,5"), and five-finger combination codes: "1,2,3,4".

[0126] Based on 31 possible output combinations and the current pitch sequence, it can be converted into fingering output for each pitch.

[0127] In this embodiment, the parameters for training the model are:

[0128] Optimizer: AdamOptimizer; Learning rate: 0.001; Gradient clipping: 5.0; Batch size: 8; Embedding matrix dimension: embeddings_dim: 128; LSTM hidden state dimension: hidden_dim: 256; Bi-LSTM hidden state dimension: hidden_dim: 256.

[0129] See Figure 5 A two-layer sequence model is used to model the fingering annotation problem, including a pitch sequence layer at the same time and a sequence layer at different times. In addition, more useful feature inputs are introduced, such as key positions, black and white key markings, and hand movement distance, which greatly improves the model performance.

[0130] In some examples, the results of the original annotation method and the annotation method of this embodiment are compared:

[0131] The sheet music with expert-annotated fingering (difficulty level 1-6) was randomly divided into a training set and a test set at a 9:1 ratio, as shown in Table 1:

[0132] Table 1

[0133] Training Collection of Musical Scores Test Collection of Musical Scores 1986 212

[0134] 1) Comparison of test set metrics for the left-hand model: The overall accuracy (consistency rate) of the single-layer sequence model (using the original annotation method) is 0.80. The metrics for each finger are shown in Table 2.

[0135] Table 2

[0136]

[0137]

[0138] The overall accuracy of the two-layer sequence model (the finger generation model of this application) is 0.91. The indicators for each finger are shown in Table 3.

[0139] Table 3

[0140] finger precision recall f1-score 1 0.96 0.96 0.96 2 0.88 0.88 0.88 3 0.89 0.87 0.88 4 0.81 0.80 0.80 5 0.94 0.96 0.95

[0141] 2) Comparison of test set metrics for the right-handed model: The overall accuracy (consistency rate) of the single-layer sequence model (using the original annotation method) is 0.77. The metrics for each finger are shown in Table 4.

[0142] Table 4

[0143] finger precision recall f1-score 1 0.84 0.88 0.86 2 0.73 0.74 0.73 3 0.73 0.71 0.72 4 0.70 0.70 0.70 5 0.87 0.82 0.85

[0144] The overall accuracy of the two-layer sequence model (the finger generation model of this application) is 0.88. The indicators for each finger are shown in Table 5.

[0145] Table 5

[0146] finger precision recall f1-score 1 0.93 0.93 0.93 2 0.87 0.85 0.86 3 0.86 0.87 0.86 4 0.82 0.83 0.82 5 0.94 0.94 0.94

[0147] In this embodiment, the fingering annotation of piano music score is modeled using a two-layer sequence model as described in this application. The result is closer to the fingering annotated by experts, and the two-layer sequence modeling method is more consistent with the representation of piano music score fingering. In addition, the introduction of more useful music score features and piano key features, such as key positions, black and white key markings, and hand movement distance, can improve the annotation effect.

[0148] The piano sheet music fingering annotation device provided in one embodiment of this application can improve the fingering annotation effect.

[0149] Figure 6 This is a schematic diagram of the structure of a piano sheet music fingering notation device provided in one embodiment of this application, as shown below. Figure 6As shown, the piano sheet music fingering annotation device of this embodiment includes: a first determining module 11, used to determine the pitch of the annotation unit in the sheet music to be annotated; wherein, the annotation unit includes a single note or a chord; a second determining module 12, used to determine the position of the key corresponding to each annotation unit on the piano and whether the key corresponding to each annotation unit is a black key or a white key, based on the pitch of each annotation unit; a third determining module 13, used to determine the distance between the key corresponding to each annotation unit in the sheet music to be annotated and the key corresponding to the previous annotation unit, based on the position of the key corresponding to each annotation unit on the piano; and a first obtaining module 14, used to determine the distance between the key corresponding to each annotation unit and the key corresponding to the previous annotation unit on the piano, based on the pitch of each annotation unit and ... position of the key corresponding to each annotation unit on the piano; and a first obtaining module 14, used to determine the distance between the key corresponding to each annotation unit and the key corresponding to the previous annotation unit on the piano, based on the position of the key corresponding to each annotation unit on the piano; and a first obtaining module 14, used to determine the distance between the key corresponding to each annotation unit and the key corresponding to the previous annotation unit on the piano, based on the position of the key corresponding to each annotation unit. The system obtains a first feature vector by considering the position of the instrument on the piano, whether the key corresponding to each marked unit is a black or white key, and the first network layer in the preset fingering generation model; a second obtaining module 15 is used to obtain a second feature vector based on the first feature vector, the distance between the key corresponding to each marked unit and the key corresponding to the previous marked unit, and the second network layer in the preset fingering generation model; a third obtaining module 16 is used to use the second feature vector as input to the conditional random field in the preset fingering generation model to obtain the fingering sequence for each marked unit; and a marking module 17 is used to mark each marked unit in the score to be marked using the obtained fingering sequence for each marked unit.

[0150] The apparatus of this embodiment can be used to perform Figure 3 The technical solutions of the method embodiments shown are similar in principle and in effect, and will not be described again here.

[0151] The apparatus of this embodiment determines the pitch of each annotation unit in the musical score to be annotated. Based on the pitch of each annotation unit, it determines the position and key characteristics of the corresponding key on the piano. Based on the position of the corresponding key on the piano, it determines the distance between the key corresponding to each annotation unit and the key corresponding to the previous annotation unit. A first feature vector is obtained based on the pitch of each annotation unit, the position of the corresponding key on the piano, whether the corresponding key is a black or white key, and the first network layer of a preset fingering generation model. A second feature vector is then obtained based on the first feature vector, the distance between the corresponding key and the previous annotation unit, and the second network layer of the preset fingering generation model. The second feature vector is used as the input to a conditional random field in the preset fingering generation model to obtain the fingering sequence for each annotation unit. The obtained fingering sequence for each annotation unit is then used... Each unit in the score to be annotated is annotated. Since the pitch of each unit, the corresponding key position, and whether it is a black or white key are all associated with its own unit, i.e., the characteristics of the score and the piano keys at the same time, are determined by combining the first network layer to form a first feature. In addition, the distance between the key corresponding to each unit and the key corresponding to the previous unit represents the characteristics of the unit at different times, and is determined by combining the second network layer to form a second feature. Therefore, in this embodiment, the pitch of the unit, the position of the key corresponding to each unit on the piano, whether the key corresponding to each unit is a black or white key, and the characteristics of the unit at different times, i.e., the distance between the keys, are input into different network layers for processing. This makes the fingering sequence of each unit more accurate, and then the obtained fingering sequence is used to annotate each unit in the score to be annotated, thereby improving the fingering annotation effect.

[0152] As an optional implementation, the second determining module is specifically used to: determine the position of the key corresponding to each marked unit in the piano and whether the key corresponding to each marked unit is a black key or a white key, based on the pitch of each marked unit and the preset correspondence between the pitch and the position of the key in the piano.

[0153] As an optional implementation, the first obtaining module includes: a determining submodule, used to determine the interval of each marked unit; and an obtaining submodule, used to obtain the first feature vector based on the pitch of each marked unit, the position of the key corresponding to each marked unit in the piano, whether the key corresponding to each marked unit is a black key or a white key, the interval of each marked unit, and the first network layer in the preset fingering generation model.

[0154] As an optional implementation, the annotation unit is a single note; wherein, the determining submodule is specifically used to: determine the interval of a single note as a preset value; and / or, the annotation unit is a chord, the chord including at least two notes; wherein, the determining submodule is specifically used to: sort the pitch of each note of the chord in descending or ascending order; the sorted pitches include a first pitch and a second pitch adjacent to and following the first pitch; subtract the second pitch from the first pitch to obtain the difference between the two, and determine the difference as the interval of the note corresponding to the second pitch.

[0155] As an optional implementation, the preset fingering generation model further includes an embedding layer; the embedding layer is connected to the first network layer and the second network layer respectively; the obtaining submodule is specifically used for: encoding the pitch of each annotation unit, the interval of each annotation unit, the position of the key corresponding to each annotation unit on the piano, and whether the key corresponding to each annotation unit is a black key or a white key according to a preset rule, and using the encoded pitch, the interval of each annotation unit, the position of the key corresponding to each annotation unit on the piano, and whether the key corresponding to each annotation unit is a black key or a white key as input to the embedding layer to obtain a first encoding vector corresponding to each annotation unit; inputting each first encoding vector into the first network layer of the preset fingering generation model to obtain a first feature vector; the second obtaining module is specifically used for: encoding the distance between the key corresponding to each annotation unit and the key corresponding to the previous annotation unit according to a preset rule, and using the encoded distance as input to the embedding layer to obtain a second encoding vector corresponding to each distance; using the first encoding vector and the second encoding vector as input to the second network layer to obtain a second feature vector.

[0156] As an optional implementation, the sheet music to be annotated includes a first annotation unit and a second annotation unit; the first annotation unit is adjacent to the second annotation unit; the first annotation unit is a single note, and the second annotation unit is a single note; the third determining module is specifically used to: determine the first position of the key corresponding to the first annotation unit on the piano, and the second position of the key corresponding to the second annotation unit on the piano; determine the distance between the key corresponding to the first annotation unit and the key corresponding to the second annotation unit based on the first position and the second position; or...

[0157] The first annotation unit is a chord, and the second annotation unit is a single note; the third determining module is specifically used for: determining the positions of the keys corresponding to each note in the first annotation unit on the piano, and the third position of the keys corresponding to the second annotation unit on the piano; determining the positions of the keys corresponding to each note in the first annotation unit on the piano, and the third position of the keys corresponding to the second annotation unit on the piano; determining the fourth position of the keys corresponding to the first annotation unit on the piano based on the positions; determining the distance between the keys corresponding to the first annotation unit and the keys corresponding to the second annotation unit based on the third and fourth positions; or...

[0158] The first annotation unit is a chord, and the second annotation unit is a chord. The step of determining the distance between the key corresponding to each annotation unit and the key corresponding to the previous annotation unit in the score to be annotated, based on the position of the key corresponding to each annotation unit on the piano, includes: determining the positions of the keys corresponding to each note in the first annotation unit and the positions of the keys corresponding to each note in the second annotation unit on the piano; determining the fifth position of the key corresponding to the first annotation unit on the piano based on the positions corresponding to the first annotation unit; determining the sixth position of the key corresponding to the second annotation unit on the piano based on the positions corresponding to the second annotation unit; and determining the distance between the key corresponding to the first annotation unit and the key corresponding to the second annotation unit based on the fifth and sixth positions.

[0159] As an optional implementation, the first network layer is a long short-term memory network; the second network layer is a bidirectional long short-term memory network; or, the first network layer is a fully self-attention network and the second network layer is a bidirectional long short-term memory network; or, the first network layer is a long short-term memory network and the second network layer is a fully self-attention network; or, the first network layer is a full attention network and the second network layer is also a full attention network.

[0160] The apparatus described in the above embodiments can be used to execute the technical solutions of the above method embodiments. The implementation principle and technical effects are similar, and will not be repeated here.

[0161] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application, as shown below. Figure 7As shown, the device may include: a housing 61, a processor 62, a memory 63, a circuit board 64, and a power supply circuit 65. The circuit board 64 is disposed inside the space enclosed by the housing 61, and the processor 62 and the memory 63 are disposed on the circuit board 64. The power supply circuit 65 is used to supply power to the various circuits or devices of the above-mentioned electronic device. The memory 63 is used to store executable program code. The processor 62 runs the program corresponding to the executable program code by reading the executable program code stored in the memory 63, and is used to execute any of the piano sheet fingering notation methods provided in the foregoing embodiments. Therefore, it can also achieve the corresponding beneficial technical effects, which have been described in detail above and will not be repeated here.

[0162] The aforementioned electronic devices exist in various forms, including but not limited to:

[0163] (1) Mobile communication devices: These devices are characterized by their mobile communication capabilities and primarily aim to provide voice and data communication. These terminals include: smartphones (e.g., iPhones), multimedia phones, feature phones, and low-end phones, etc.

[0164] (2) Ultra-mobile personal computer devices: These devices fall under the category of personal computers, possessing computing and processing capabilities, and generally also feature mobile internet access. These terminals include PDAs, MIDs, and UMPCs, such as the iPad.

[0165] (3) Portable entertainment devices: These devices can display and play multimedia content. This category includes: audio and video players (such as iPods), handheld game consoles, e-books, as well as smart toys and portable car navigation devices.

[0166] (4) Server: A device that provides computing services. The components of a server include a processor, hard disk, memory, system bus, etc. Servers are similar to general computer architectures, but because they need to provide highly reliable services, they have higher requirements in terms of processing power, stability, reliability, security, scalability, and manageability.

[0167] (5) Other electronic devices with data interaction functions.

[0168] Accordingly, embodiments of this application also provide a computer-readable storage medium storing one or more programs, which can be executed by one or more processors to implement any of the piano sheet fingering notation methods provided in the foregoing embodiments, thus achieving the corresponding technical effects. This has been described in detail above and will not be repeated here.

[0169] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0170] The various embodiments in this specification are described in a related manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments.

[0171] In particular, the device embodiment is basically similar to the method embodiment, so the description is relatively simple. For relevant details, please refer to the description of the method embodiment.

[0172] For ease of description, the above apparatus is described by dividing it into various functional units / modules. Of course, in implementing this application, the functions of each unit / module can be implemented in one or more software and / or hardware.

[0173] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0174] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for noting fingering in piano sheet music, characterized in that, include: Determine the pitch of the unit to be annotated in the musical score; wherein, the unit to be annotated includes a single note or a chord; Based on the pitch of each marked unit, determine the position of the key on the piano corresponding to each marked unit and whether the key corresponding to each marked unit is a black key or a white key; Based on the position of the key corresponding to each annotation unit in the piano, determine the distance between the key corresponding to each annotation unit in the score to be annotated and the key corresponding to the previous annotation unit; The first feature vector is obtained based on the pitch of each labeled unit, the position of the key corresponding to each labeled unit on the piano, whether the key corresponding to each labeled unit is a black key or a white key, and the first network layer in the preset fingering generation model. The second feature vector is obtained based on the first feature vector, the distance between the key corresponding to each labeled unit and the key corresponding to the previous labeled unit, and the second network layer in the preset fingering generation model. The second feature vector is used as the input to the conditional random field in the preset fingering generation model to obtain the fingering sequence for each labeled unit. Using the fingering sequence of each obtained annotation unit, annotate each annotation unit in the score to be annotated.

2. The method according to claim 1, characterized in that, The process of determining the position of the key on the piano corresponding to each marked unit and whether the key corresponding to each marked unit is a black key or a white key, based on the pitch of each marked unit, includes: Based on the pitch of each marked unit and the preset correspondence between the pitch and the position of the key on the piano, the position of the key corresponding to each marked unit on the piano and whether the key corresponding to each marked unit is a black key or a white key are determined.

3. The method according to claim 1, characterized in that, The first feature vector is obtained based on the pitch of each labeled unit, the position of the corresponding key on the piano, whether the key corresponding to each labeled unit is a black or white key, and the first network layer in the preset fingering generation model. This feature vector includes: Determine the interval for each marked unit; The first feature vector is obtained based on the pitch of each labeled unit, the position of the key corresponding to each labeled unit on the piano, whether the key corresponding to each labeled unit is a black key or a white key, the interval of each labeled unit, and the first network layer in the preset fingering generation model.

4. The method according to claim 3, characterized in that, The unit of measurement is a single tone; The determination of the interval for each marked unit includes: Set the interval of a single note to a preset value; and / or, The unit of measurement is a chord, and a chord includes at least two notes; The determination of the interval for each marked unit includes: The pitches of each note in the chord are sorted in descending or ascending order; the sorted pitches include a first pitch and a second pitch that is adjacent to and follows the first pitch. The difference between the second pitch and the first pitch is obtained by subtracting the second pitch from the first pitch, and this difference is determined as the interval of the note corresponding to the second pitch.

5. The method according to claim 3, characterized in that, The preset fingering generation model further includes an embedding layer; the embedding layer is connected to the first network layer and the second network layer respectively; The first feature vector is obtained based on the pitch of each labeled unit, the position of the corresponding key on the piano, whether the corresponding key is a black or white key, the interval of each labeled unit, and the first network layer in the preset fingering generation model, including: The pitch, interval, position of the corresponding key on the piano, and whether the corresponding key is a black or white key for each labeled unit are encoded according to a preset rule. The encoded pitch, interval, position of the corresponding key on the piano, and whether the corresponding key is a black or white key are used as input to the embedding layer to obtain the first encoding vector corresponding to each labeled unit. Each of the first encoded vectors is input into the first network layer of a preset finger generation model to obtain the first feature vector; The step of obtaining the second feature vector based on the first feature vector, the distance between the key corresponding to each labeled unit and the key corresponding to the previous labeled unit, and the second network layer in the preset fingering generation model includes: The distance between the key corresponding to each labeled unit and the key corresponding to the previous labeled unit is encoded according to a preset rule, and the encoded distance is used as the input of the embedding layer to obtain the second encoded vector corresponding to each distance; The first encoding vector and the second encoding vector are used as inputs to the second network layer to obtain the second feature vector.

6. The method according to claim 1, characterized in that, The musical score to be annotated includes a first annotation unit and a second annotation unit; the first annotation unit and the second annotation unit are adjacent to each other; The first annotation unit is a single note, and the second annotation unit is a single note; determining the distance between the key corresponding to each annotation unit and the key corresponding to the previous annotation unit in the score to be annotated, based on the position of the key corresponding to each annotation unit on the piano, includes: Determine the first position of the key in the piano corresponding to the first marked unit, and the second position of the key in the piano corresponding to the second marked unit; Based on the first position and the second position, determine the distance between the piano key corresponding to the first labeled unit and the piano key corresponding to the second labeled unit; or, The first unit of annotation is a chord, and the second unit of annotation is a single note; The step of determining the distance between the key corresponding to each marked unit and the key corresponding to the previous marked unit in the score to be annotated, based on the position of the key corresponding to each marked unit on the piano, includes: Determine the positions of the keys corresponding to each note in the first annotation unit on the piano, and the third position of the key corresponding to the second annotation unit on the piano; Based on the aforementioned positions, determine the fourth position of the key corresponding to the first marked unit on the piano; Based on the third and fourth positions, determine the distance between the key corresponding to the first labeled unit and the key corresponding to the second labeled unit; or... The first unit of annotation is chord, and the second unit of annotation is chord; The step of determining the distance between the key corresponding to each marked unit and the key corresponding to the previous marked unit in the score to be annotated, based on the position of the key corresponding to each marked unit on the piano, includes: Determine the positions of the keys corresponding to each note in the first annotation unit on the piano, and the positions of the keys corresponding to each note in the second annotation unit on the piano. Based on the positions corresponding to the first marked unit, determine the fifth position of the piano key corresponding to the first marked unit in the piano; Based on the positions corresponding to the second annotation unit, determine the sixth position of the key on the piano corresponding to the second annotation unit; Based on the fifth position and the sixth position, the distance between the piano key corresponding to the first annotation unit and the piano key corresponding to the second annotation unit is determined.

7. The method according to claim 1, characterized in that, The first network layer is a long short-term memory network; the second network layer is a bidirectional long short-term memory network; or, the first network layer is a fully self-attention network and the second network layer is a bidirectional long short-term memory network; or, the first network layer is a long short-term memory network and the second network layer is a fully self-attention network; or, the first network layer is a fully attention network and the second network layer is also a fully attention network.

8. A piano sheet music fingering marking device, characterized in that, include: The first determining module is used to determine the pitch of the marking unit in the musical score to be annotated; wherein, the marking unit includes a single note or a chord; The second determining module is used to determine the position of the key in the piano corresponding to each marked unit and whether the key corresponding to each marked unit is a black key or a white key, based on the pitch of each marked unit. The third determining module is used to determine the distance between the key corresponding to each marked unit in the score to be marked and the key corresponding to the previous marked unit, based on the position of the key corresponding to each marked unit in the piano. The first module is used to obtain the first feature vector based on the pitch of each labeled unit, the position of the key corresponding to each labeled unit in the piano, whether the key corresponding to each labeled unit is a black key or a white key, and the first network layer in the preset fingering generation model. The second obtaining module is used to obtain the second feature vector based on the first feature vector, the distance between the piano key corresponding to each labeled unit and the piano key corresponding to the previous labeled unit, and the second network layer in the preset fingering generation model. The third module is used to take the second feature vector as the input of the conditional random field in the preset finger generation model to obtain the finger sequence for each labeled unit. The annotation module is used to annotate each annotation unit in the score to be annotated using the fingering sequence of each obtained annotation unit.

9. The apparatus according to claim 8, characterized in that, The second determining module is specifically used for: Based on the pitch of each marked unit and the preset correspondence between the pitch and the position of the key on the piano, the position of the key corresponding to each marked unit on the piano and whether the key corresponding to each marked unit is a black key or a white key are determined.

10. The apparatus according to claim 8, characterized in that, The first obtaining module includes: The determination submodule is used to determine the interval for each marked unit; A submodule is obtained to obtain the first feature vector based on the pitch of each labeled unit, the position of the key corresponding to each labeled unit on the piano, whether the key corresponding to each labeled unit is a black key or a white key, the interval of each labeled unit, and the first network layer in the preset fingering generation model.

11. The apparatus according to claim 10, characterized in that, The unit of measurement is a single tone; Specifically, the determining submodule is used for: Set the interval of a single note to a preset value; and / or, The unit of measurement is a chord, and a chord includes at least two notes; Specifically, the determining submodule is used for: The pitches of each note in the chord are sorted in descending or ascending order; the sorted pitches include a first pitch and a second pitch that is adjacent to and follows the first pitch. The difference between the second pitch and the first pitch is obtained by subtracting the second pitch from the first pitch, and this difference is determined as the interval of the note corresponding to the second pitch.

12. The apparatus according to claim 10, characterized in that, The preset fingering generation model further includes an embedding layer; the embedding layer is connected to the first network layer and the second network layer respectively; The obtained submodule is specifically used for: The pitch, interval, position of the corresponding key on the piano, and whether the corresponding key is a black or white key for each labeled unit are encoded according to a preset rule. The encoded pitch, interval, position of the corresponding key on the piano, and whether the corresponding key is a black or white key are used as input to the embedding layer to obtain the first encoding vector corresponding to each labeled unit. Each of the first encoded vectors is input into the first network layer of a preset finger generation model to obtain the first feature vector; The second obtaining module is specifically used for: The distance between the key corresponding to each labeled unit and the key corresponding to the previous labeled unit is encoded according to a preset rule, and the encoded distance is used as the input of the embedding layer to obtain the second encoded vector corresponding to each distance; The first encoding vector and the second encoding vector are used as inputs to the second network layer to obtain the second feature vector.

13. The apparatus according to claim 8, characterized in that, The musical score to be annotated includes a first annotation unit and a second annotation unit; the first annotation unit and the second annotation unit are adjacent to each other; The first annotation unit is a single note, and the second annotation unit is a single note; the third determining module is specifically used for: Determine the first position of the key in the piano corresponding to the first marked unit, and the second position of the key in the piano corresponding to the second marked unit; Based on the first position and the second position, determine the distance between the key corresponding to the first labeled unit and the key corresponding to the second labeled unit; or... The first unit of annotation is a chord, and the second unit of annotation is a single note; the third determining module is specifically used for: Determine the positions of the keys corresponding to each note in the first annotation unit on the piano, and the third position of the key corresponding to the second annotation unit on the piano; Determine the positions of the keys corresponding to each note in the first annotation unit on the piano, and the third position of the key corresponding to the second annotation unit on the piano; Based on the aforementioned positions, determine the fourth position of the key corresponding to the first marked unit on the piano; Based on the third and fourth positions, determine the distance between the key corresponding to the first labeled unit and the key corresponding to the second labeled unit; or... The first unit of annotation is chord, and the second unit of annotation is chord; The step of determining the distance between the key corresponding to each marked unit and the key corresponding to the previous marked unit in the score to be annotated, based on the position of the key corresponding to each marked unit on the piano, includes: Determine the positions of the keys corresponding to each note in the first annotation unit on the piano, and the positions of the keys corresponding to each note in the second annotation unit on the piano. Based on the positions corresponding to the first marked unit, determine the fifth position of the piano key corresponding to the first marked unit in the piano; Based on the positions corresponding to the second annotation unit, determine the sixth position of the key on the piano corresponding to the second annotation unit; Based on the fifth position and the sixth position, the distance between the piano key corresponding to the first annotation unit and the piano key corresponding to the second annotation unit is determined.

14. The apparatus according to claim 8, characterized in that, The first network layer is a long short-term memory network; the second network layer is a bidirectional long short-term memory network; or, the first network layer is a fully self-attention network and the second network layer is a bidirectional long short-term memory network; or, the first network layer is a long short-term memory network and the second network layer is a fully self-attention network; or, the first network layer is a fully attention network and the second network layer is also a fully attention network.

15. An electronic device, characterized in that, The electronic device includes: a housing, a processor, a memory, a circuit board, and a power supply circuit, wherein the circuit board is disposed inside the space enclosed by the housing, and the processor and the memory are disposed on the circuit board; the power supply circuit is used to supply power to various circuits or devices of the electronic device; the memory is used to store executable program code; the processor runs a program corresponding to the executable program code by reading the executable program code stored in the memory, for executing the piano score fingering notation method according to any one of claims 1-7.

16. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores one or more programs, which can be executed by one or more processors to implement the piano score fingering notation method according to any one of claims 1-7.

Images