A melody skeleton extraction method based on rhythm and pitch stability

Abstract

The present invention discloses a melody skeleton extraction method based on rhythm and pitch stability, comprising the following steps: (1) receiving a melody information sequence and performing note quantization using an adaptive hybrid analysis rate quantization method; (2) classifying the notes into long fortissimo, beat accent, syncopation, and modifier based on the attributes of each note according to the quantized melody information sequence; (3) designing a rhythm node determination rule based on the note classification, and extracting the rhythm node of each melody according to the determination rule; (4) dividing the melody motive boundary into groups of two or three notes based on the extracted rhythm nodes; and (5) calculating the backbone tones in the melody motive boundary based on a pitch spiral array. Utilizing the present invention, it is possible to extract musical elements that can attract the audience's attention based on musical acoustic cues.

Description

Technical Field

[0001] The present invention relates to the field of music technology, and in particular to a melody skeleton extraction method based on rhythm and pitch stability. Background Art

[0002] Algorithmic composition refers to the use of computers to create music automatically. The main methods include Markov chains, musical grammar, genetic algorithms, artificial neural networks, etc.

[0003] With the emergence of machine learning algorithms and the expansion of digital music resources, intelligent composition technology has emerged. Computer-assisted learning has replaced human-set rules and become the mainstream method of music generation.

[0004] The intelligent composition process encompasses multiple composition tasks, such as main melody generation, accompaniment generation and continuation, and melody generation based on lyrics. Main melody generation, a crucial component of music generation, has garnered significant research attention. Numerous deep neural network-based methods have been proposed to enable machines to acquire melody-writing knowledge or skills from data without human intervention.

[0005] The current mainstream methods of using deep learning to generate music can be divided into two types: the first one relies entirely on the performance of the model itself to generate music. They often focus on a certain attribute of the model, such as the paper published at the ICLR conference in 2018: Music Transformer: Generating Music With Long-term Structure. The author proposed the memory problem of transformer when processing music data, which makes it difficult for the generated music to have a long-term dependency structure. Among them, to address this problem, the author modified the attention mechanism and proposed an algorithm to reduce the memory requirement of the model to the linear length of the sequence, so that the model can handle longer long-term dependency structures.

[0006] The second is a knowledge-enhanced deep learning model that incorporates knowledge. For music generation tasks, knowledge can go beyond the current input content, reduce the number of training samples, and generate richer music. For example, in the paper published in the journal Artificial Intelligence in 2020: PopMNet: generating structured Pop musicmelodies using Neural Network, the authors emphasized that in the field of music generation, automatically generating the long-term structure of music is still a challenge. Among them, the authors designed a method to first generate music structure and then generate specific music, including: using paired bars to define the music structure, and dividing their relationship into two types of repetition and development, extracting midi data and music structure, and inputting the PopMNet model. The model includes a structure generation network based on a convolutional neural network and a melody generation network based on a recurrent neural network. The two networks are trained separately and finally output music.

[0007] In recent years, researchers have increasingly favored methods for incorporating knowledge into models in intelligent music. However, the knowledge incorporated into most models is often superficial musical features. While these surface musical features are crucial for music analysis, simply extracting them only yields isolated elements, and treating them equally is not enough.

[0008] Music is an organic combination of musical elements, each of which has relative importance. Based on Schenk's theory, we can hierarchically reduce music to its basic elements, forming a tree-like structure in which each musical element is explained. However, current work only analyzes elements without analyzing the relationships between them, neglecting the deeper structural information beneath the surface features of music. This makes it difficult for the generated music to have organic coherence. Summary of the Invention

[0009] The present invention provides a melody skeleton extraction method based on rhythm and pitch stability, which can extract musical elements that can attract the audience's attention according to music acoustic clues.

[0010] A melody skeleton extraction method based on rhythm and pitch stability comprises the following steps:

[0011] (1) receiving a melody information sequence and performing note quantization using an adaptive hybrid analysis note quantization method;

[0012] (2) Based on the quantized melody information sequence, the notes are classified into long fortissimo, beat accent, syncopation and modifier according to the attributes of each note;

[0013] (3) Designing a rule for determining rhythmic nodes based on the classification of notes, and extracting the rhythmic nodes of each melody according to the rule;

[0014] (4) Based on the extracted rhythmic nodes, the boundaries of the melodic motive are divided into groups of two or three notes;

[0015] (5) Based on the pitch spiral array, calculate the backbone tones in the melodic motive boundary.

[0016] This paper analyzes single-melody music into two components: backbone notes and ornaments. These components represent the music's stabilizing and decorative elements, respectively. This method of extracting backbone notes from a large number of songs enables knowledge-enhanced deep learning models to better understand the mapping between human subjective hearing and musical elements, thereby improving the quality of music generation.

[0017] The classification of "core" and "ornament" implies a hierarchical approach to musical analysis, whereby "core" and "ornament" exist in a subordinate relationship. Unlike traditional analysis, which tends to treat musical elements as isolated, the "core" and "ornament" approach emphasizes the organic connections between musical elements and is more consistent with the subjective experience of human musical listening.

[0018] In music psychology, research on how the mind organizes musical information mainly focuses on two aspects: Gestalt principles and musical syntax cognition.

[0019] Gestalt psychology is a key theoretical underpinning of music psychology, providing a valuable starting point for understanding how the human mind organizes musical information. Gestalt theory's fundamental premise is that "the whole is greater than the sum of its parts," emphasizing the organic connections between elements. The object-ground relationship explains why some musical elements are more important than others when we listen to music. The principles of Gestalt tendency, proximity, similarity, common direction, and parsimony connect isolated musical elements.

[0020] The study of musical syntax cognition further elucidates the application of Gestalt psychology to musical understanding. From Schenck's musical analysis to Ledahl's Generative Theory of Tonal Music (GTTM), both demonstrate the existence of musical syntax. Through syntax, the various parts of music are connected and form a meaningful whole. Therefore, syntax embodies the structural rules of music. Furthermore, the existence of musical syntax is a prerequisite for human understanding of music. It is the prerequisite for individuals to grasp the overall and even deeper meaning of music. It is particularly important in understanding the meaning of music and even determines the success or failure of individual musical appreciation.

[0021] Therefore, the present invention emphasizes the addition of psychological content to the knowledge extraction process, so that the deep learning model that incorporates knowledge can understand the basic rules of musical syntax, thereby making the generated music more in line with the basic needs of human musical aesthetics.

[0022] In step (1), when performing note quantization, notes shorter than 64th notes are discarded, and notes longer than one measure are saved as whole notes; and the remaining notes are divided into straight notes and triplets according to the duration and time symbol of the notes;

[0023] Among them, two or three consecutive notes with a duration error of no more than plus or minus 30 ticks and the same duration of the triplets are considered triplets.

[0024] Once the notes are divided into straight notes and triplets, select a quantization grid for each note, specifically:

[0025] When the note duration is longer than or equal to 16th note, the starting point of the note is aligned with its nearest 16th note time grid;

[0026] When the note duration is between 16th notes and 32nd notes, the starting point of the note is aligned with its nearest 32nd note time grid;

[0027] When the note duration is between 32nd notes and 64th notes, the starting point of the note is aligned with its nearest 64th note time grid;

[0028] Additionally, all note offsets are aligned to the 64th note time grid, and in the case of triplets, the note attack and offset times are aligned to the 48th note time grid.

[0029] In step (2), the recognition rule of long strong tone is:

[0030] Create a long note dictionary and obtain the index of one or more notes with the longest duration in the measure by comparing the duration of the notes in the measure.

[0031] The rules for identifying beat accents are:

[0032] Simplify the measure into a 2-beat accent unit or a 3-beat accent unit; the first accent that appears is the strong beat, and the subsequent accents are the secondary strong beats;

[0033] When the starting position of a note is on the first or third beat of a measure, the note is added to the beat accent set;

[0034] When a syncopation occurs within three 16th-note durations after a beat accent, and the duration of the syncopation is greater than the duration of the beat accent, the beat accent is removed from the beat accent set.

[0035] In a 16th-note beat grid, syncopation is determined by the attack position of the note. The specific rules are as follows:

[0036] The type of syncopation used is determined by the position of the attack of the notes in the measure in the 16th note beat grid:

[0037] If the strong and weak beats are divided by quarter notes, then the weak beat positions are: grid5, grid13; when the note starts at the above positions, the strong and weak beat positions of the quarter note are used to determine the syncopation;

[0038] If the eighth note is used to divide the strong and weak beats, then the weak beat positions are: grid3, grid7, grid11, grid15; when the note starts at the above positions, the strong and weak beat positions of the eighth note are used to determine the syncopation;

[0039] If the strong and weak beat positions are divided by sixteenth notes, then the weak beat positions are: grid2, grid4, grid6, grid8, grid10, grid12, grid14, grid16; when the note starts at the above positions, the strong and weak beat position division rules of sixteenth notes are used to determine the syncopation;

[0040] When a note starts from its corresponding weak beat position and ends at the next corresponding strong beat or secondary strong beat position, the note is added to the syncopated set.

[0041] In step (3), the rules for determining rhythmic nodes are designed based on the classification of notes as follows:

[0042] According to the arrangement and combination of long fortissimo, beat accent and syncopation, there are 8 arrangements: (1,1,1), (1,0,1), (1,0,0), (1,1,0), (0,1,1), (0,1,0), (0,0,1) and (0,0,0), and the three arrangements (1,0,0), (1,1,0) and (0,1,1) are judged as rhythmic nodes.

[0043] The specific process of step (4) is:

[0044] (4-1) Identify the rhythmic nodes in the main melody as the beginning or end of a rhythmic unit;

[0045] When other notes appear before a rhythmic node: After a rhythmic node appears, the other notes that appeared before and this rhythmic node are set as a rhythmic unit;

[0046] When other notes appear after a rhythmic node: before the next rhythmic node appears, the rhythmic node and other notes are set as a rhythmic unit;

[0047] (4-2) Within a rhythmic unit, only rests of 240 ticks or less are allowed. When the rest is greater than or equal to 240 ticks, the rest is skipped until the next note appears, and recognition begins with a new rhythmic unit.

[0048] (4-3) Traverse each rhythmic unit segment, taking two or three notes as a unit, and calculate the optimal solution for the melody motive boundary by traversing all possible combinations of two and three notes;

[0049] (4-4) Output the melodic motive boundary.

[0050] The specific process of step (5) is:

[0051] (5-1) Using a mode and tonality recognition algorithm to identify the global tonality of the melody and determine the tonic of the melody;

[0052] (5-2) Identify the effective center of the melody based on the pitch spiral array and the main tone;

[0053] (5-3) Based on the melody motive boundary and the effect center, use the pitch spiral array to compare the Euclidean distance between each note within the melody motive boundary and the effect center;

[0054] (5-4) Based on the Euclidean distance between each note within the melody motive boundary and the effect center, the nearest note is extracted as the backbone tone.

[0055] Compared with the prior art, the present invention has the following beneficial effects:

[0056] 1. Let the knowledge-infused deep learning model understand the basic rules of musical syntax, so that the generated music is more in line with the basic needs of human musical aesthetics.

[0057] 2. The music generated under the "backbone" framework is more robust.

[0058] 3. The hierarchical analysis method and the hierarchical generation method it implies provide space for users' personalized needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 This is a flow chart of a melody skeleton extraction method based on rhythm and pitch stability of the present invention;

[0060] Figure 2 It is the relationship between strong and weak within a measure at different note granularity;

[0061] Figure 3 It is a combination of possible rhythmic nodes among the permutations and combinations of the three note types;

[0062] Figure 4 This is a schematic diagram of the note arrangement of the pitch spiral array and the effect center identification method. DETAILED DESCRIPTION

[0063] The present invention will be described in further detail below with reference to the accompanying drawings and examples. It should be noted that the following examples are intended to facilitate understanding of the present invention and do not have any limiting effect on the present invention.

[0064] like Figure 1 As shown, a melody skeleton extraction method based on rhythm and pitch stability includes the following steps:

[0065] Step 1: Note quantization.

[0066] Receive a sequence of melody information and discard notes shorter than a 64th note, while keeping notes longer than a measure as whole notes.

[0067] The remaining notes are divided into straight notes and triplets based on their duration and time signature. Triplet durations do not always have to be three notes. It is common to have only two notes in a triplet, as in Swing. Furthermore, each note in a triplet theoretically has equal rhythmic value, but in practice, most notes differ slightly from each other in musical performance. Therefore, two or three consecutive notes with roughly the same duration that matches the duration of a triplet are considered triplets. In this work, based on reasonable experimental statistical results, a 20% difference between the actual triplet duration and the standard triplet duration is allowed.

[0068] Automatically selects a suitable quantization grid for each note.

[0069] When the note duration is longer than or equal to 16th note, the starting point of the note will be aligned with its nearest 16th note time grid;

[0070] When the note duration is between 16th notes and 32nd notes, the starting point of the note will be aligned with its nearest 32nd note time grid;

[0071] When the note duration is between 32nd notes and 64th notes, the note start point will be aligned with the nearest 64th note time grid.

[0072] Additionally, all note offsets are aligned to the 64th note time grid. In the case of triplets, the note attack and offset times are aligned to the 48th note time grid.

[0073] Step 2: Based on the quantized melody information sequence, classify the notes into long fortissimo, beat accent, syncopation, and ornament according to the attributes of each note.

[0074] Agogic identification: The range of long notes to be compared is set within a measure. In some cases, there may be more than one longest note within the range. However, the difference in their importance can still be determined by their position. Notes on the beat accent are far more important than notes at other beat positions. Therefore, in this case, this rule can be used to compare long notes within the range and minimize the number of long notes. The specific rules are as follows:

[0075] 1. Create a long sound dictionary

[0076] 2. By comparing the duration of the notes in the measure, obtain the index of one or more notes with the longest duration in the measure.

[0077] Metrical accent recognition: No matter how long a measure is, it can be simplified into a 2-beat accent unit, or a 3-beat accent unit. The first accent that appears is the strong beat, and the accent that appears later is the secondary strong beat. The processed MIDI data are all 4 / 4 beats (1 / 4 note as one beat, four beats per measure), and the strong and weak rules are "strong then weak, secondary strong then weak". Since the structural line notes are the notes that play a stabilizing role in Schenk analysis, the notes on the strong beat or secondary strong beat are more likely to be rhythmic nodes, such as Figure 2 shown.

[0078] However, when syncopation occurs, the normal rhythmic pattern of beat strength will change, and the position of some beat accents will be shifted. In this case, the first and third beats cannot be directly extracted as beat accents. Specifically, when a syncopation occurs immediately after a beat accent, the accent position is likely to shift to the syncopation. Therefore, the following rules are used to extract the beat accent:

[0079] When the starting position of a note is on the first or third beat of a measure, the note is added to the beat accent set.

[0080] When a syncopation occurs within three 16th-note durations after a beat accent, and the duration of the syncopation is greater than the duration of the beat accent, the beat accent is removed from the beat accent set.

[0081] Syncopation: In a 16th-note beat grid, syncopation can be determined based on the attack position of the note. The specific rules are as follows:

[0082] The syncopation type to be used is determined by the position of the attack (Strat) of the notes in the measure in the 16th note beat network.

[0083] If the strong and weak beat positions are divided by quarter notes, then the weak beat positions are: grid5, grid13; when the note starts at the above positions, the strong and weak beat position division rules of quarter notes are used to determine the syncopation.

[0084] If the strong and weak beat positions are divided by eighth notes, then the weak beat positions are: grid3, grid7, grid11, grid15; when the note starts at the above positions, the strong and weak beat position division rules of eighth notes are used to determine the syncopation.

[0085] If the strong and weak beat positions are divided by sixteenth notes, then the weak beat positions are: grid2, grid4, grid6, grid8, grid610, grid12, grid14, grid16; when the note starts at the above positions, the strong and weak beat position division rules of sixteenth notes are used to determine the syncopation.

[0086] When a note starts from its corresponding weak beat position and ends at the next corresponding strong beat or secondary strong beat position, the note is added to the syncopated set.

[0087] Step 3: Prosodic node identification.

[0088] After completing the identification of the types of all notes in a single melody, the rhythmic node is judged according to the following rules. All possibilities are summarized as follows: Figure 3 As shown in the figure, according to the arrangement and combination of long fortissimo, beat accent and syncopation, there are 8 arrangements: (1,1,1), (1,0,1), (1,0,0), (1,1,0), (0,1,1), (0,1,0), (0,0,1) and (0,0,0), and the three arrangements (1,0,0), (1,1,0) and (0,1,1) are judged as rhythmic nodes.

[0089] Step 4: Based on the extracted rhythmic nodes, divide the melodic motive boundaries into groups of two or three notes.

[0090] Identify the rhythmic boundary notes in the main melody as the beginning or end of a rhythmic unit.

[0091] When other notes appear before a rhythmic node: After a rhythmic node appears, other notes that appeared before it and this rhythmic node are set as a rhythmic unit.

[0092] When other notes appear after a rhythmic node: Before the next rhythmic node appears, the rhythmic node and other notes that appear are set as one rhythmic unit.

[0093] Within a rhythmic unit, only rests of 240 ticks or less are allowed. When the rest is greater than or equal to 240 ticks, the rest is skipped until the next note appears, and recognition begins with a new rhythmic unit.

[0094] Traverse each rhythmic unit segment, taking two or three notes as a unit, and calculate the optimal solution for the melodic motif boundary by traversing all possible combinations of two or three notes. Output the melodic motif boundary.

[0095] Step 5: Based on the pitch spiral array, calculate the backbone tones in the melodic motive boundary.

[0096] Based on the mode and tonality recognition algorithm, the global tonality of the melody is identified and the tonic of the melody is determined; based on the pitch spiral array and the tonic, the effect center of the melody is identified; based on the melody motive boundary and the effect center, the pitch spiral array is used to compare the Euclidean distance of each note within the boundary from the effect center; based on the Euclidean distance of each note within the melody motive boundary and the effect center, the closest note is extracted as the backbone tone.

[0097] like Figure 4 As shown, the five-pointed star in the figure is the effect center, which is determined by the midpoint of the three planes formed by the chord tones of the three positive triads in the tonality, and then by the midpoint of the triangle formed by these three midpoints.

[0098] The embodiments described above provide a detailed description of the technical solutions and beneficial effects of the present invention. It should be understood that the above are only specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, supplements and equivalent substitutions made within the scope of the principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A melody skeleton extraction method based on rhythm and pitch stability, characterized in that: The following steps are involved: (1) receiving a melody information sequence and performing note quantization using an adaptive hybrid analysis note quantization method; When performing note quantization, notes shorter than 64th notes are discarded, and notes longer than one measure are kept as whole notes. The remaining notes are divided into straight notes and triplets based on their duration and time signature. Two or three consecutive notes with a duration error of no more than plus or minus 30 ticks and consistent with the duration of a triplet are considered triplets. After the notes are divided into straight notes and triplets, a quantization grid is selected for each note. Specifically, when the note duration is longer than or equal to 16th notes, the starting point of the note is aligned with its nearest 16th note time grid. When the note duration is between 16th notes and 32nd notes, the starting point of the note is aligned with its nearest 32nd note time grid; When the note duration is between 32nd notes and 64th notes, the starting point of the note is aligned with its nearest 64th note time grid; Furthermore, all note offsets are aligned to the 64th note time grid, and in the case of triplets, the note attack and offset times are aligned to the 48th note time grid; (2) Based on the quantized melodic information sequence, the notes are classified into long fortissimo, beat accent, syncopation and modifier according to the attributes of each note; (3) Design a rule for determining rhythmic nodes based on the classification of notes, and extract the rhythmic nodes of each melody according to the rule; the specific determination rule is: According to the arrangement and combination of long fortissimo, beat accent, and syncopation, there are 8 arrangements: (1,1,1), (1,0,1), (1,0,0), (1,1,0), (0,1,1), (0,1,0), (0,0,1), and (0,0,0). Among them, the three arrangements (1,0,0), (1,1,0), and (0,1,1) are identified as rhythmic nodes. (4) Based on the extracted rhythmic nodes, the boundaries of the melodic motive are divided into groups of two or three notes; (5) Based on the pitch spiral array, calculate the backbone tones in the melodic motive boundary.

2. The melody skeleton extraction method based on rhythm and pitch stability according to claim 1, wherein In step (2), the recognition rule of long fortissimo is: Create a long note dictionary and obtain the index of one or more notes with the longest duration in the measure by comparing the duration of the notes in the measure.

3. The melody skeleton extraction method based on rhythm and pitch stability according to claim 1, wherein In step (2), the recognition rule of beat accent is: Simplify the measure into a 2-beat accent unit or a 3-beat accent unit; the first accent that appears is the strong beat, and the subsequent accents are the secondary strong beats; When the starting position of a note is on the first or third beat of a measure, the note is added to the beat accent set; When a syncopation occurs within three 16th-note durations after a beat accent, and the duration of the syncopation is greater than the duration of the beat accent, the beat accent is removed from the beat accent set.

4. The melody skeleton extraction method based on rhythm and pitch stability according to claim 1, wherein In step (2), syncopation is determined based on the onset position of the note in the sixteenth-note beat grid. The specific rules are as follows: The type of syncopation used is determined by the position of the attack of the notes in the measure in the 16th note beat grid: If the strong and weak beats are divided by quarter notes, then the weak beat positions are: grid5, grid13; when the note starts at the above positions, the strong and weak beat positions of the quarter note are used to determine the syncopation; If the eighth note is used to divide the strong and weak beats, then the weak beat positions are: grid3, grid7, grid11, grid15; when the note starts at the above positions, the strong and weak beat positions of the eighth note are used to determine the syncopation; If the strong and weak beat positions are divided by sixteenth notes, then the weak beat positions are: grid2, grid4, grid6, grid8, grid610, grid12, grid14, grid16; when the note starts at the above positions, the strong and weak beat position division rules of sixteenth notes are used to determine the syncopation; When a note starts from its corresponding weak beat position and ends at the next corresponding strong beat or secondary strong beat position, the note is added to the syncopated set.

5. The melody skeleton extraction method based on rhythm and pitch stability according to claim 1, wherein The specific process of step (4) is: (4-1) Identify the rhythmic nodes in the main melody as the beginning or end of a rhythmic unit; When other notes appear before a rhythmic node: After a rhythmic node appears, the other notes that appeared before and this rhythmic node are set as a rhythmic unit; When other notes appear after a rhythmic node: before the next rhythmic node appears, the rhythmic node and other notes are set as a rhythmic unit; (4-2) Within a rhythmic unit, only rests of 240 ticks or less are allowed. When the rest is greater than or equal to 240 ticks, the rest is skipped until the next note appears, and recognition begins with a new rhythmic unit. (4-3) Traverse each rhythmic unit segment, taking two or three notes as a unit, and calculate the optimal solution for the melody motive boundary by traversing all possible combinations of two and three notes; (4-4) Output the melodic motive boundary.

6. The melody skeleton extraction method based on rhythm and pitch stability according to claim 1, wherein The specific process of step (5) is: (5-1) Using a mode and tonality recognition algorithm to identify the global tonality of the melody and determine the tonic of the melody; (5-2) Identify the effective center of the melody based on the pitch spiral array and the main tone; (5-3) Based on the melody motive boundary and the effect center, use the pitch spiral array to compare the Euclidean distance between each note within the melody motive boundary and the effect center; (5-4) Based on the Euclidean distance between each note within the melody motive boundary and the effect center, the nearest note is extracted as the backbone tone.

Images