Automatic multi-feature labeling method and device for folk music score

Abstract

The invention provides an automatic multi-feature labeling method and device for folk music scores, and relates to the technical field of sound analysis, and the method comprises the steps: obtaining a target folk music score corpus; carrying out symbol analysis and time-frequency transformation, and respectively extracting a note sequence matrix and a spectrum feature matrix; constructing an initial feature embedding space through a variational auto-encoder; hierarchical labeling labels are called, wherein the hierarchical labeling labels comprise pitch contours, rhythm structures, decorative sound positions and types and force trend curves; training an initial annotation model, wherein the initial annotation model is a Transforme r network based on multi-head self-attention; performing increment fine adjustment by using an elastic weight integration strategy to generate a target annotation model; and outputting a multi-feature annotation result of the target folk music score. The technical problem that in the prior art, music data is processed mainly depending on manual annotation and a traditional feature extraction method, and diversity features in music scores cannot be comprehensively captured, so that the annotation process is low in efficiency and inaccurate is solved.

Description

Technical Field

[0001] The present invention relates to the technical field of sound analysis, and in particular to an automated multi-feature annotation method and device for folk music scores. Background Art

[0002] Folk music scores contain numerous complex features, and these features vary significantly across different performance styles, instruments, and notation categories. This presents numerous challenges for traditional music annotation methods, such as how to unify standards across diverse scores and performance styles, how to cope with frequently changing rhythms and dynamics, and how to handle the unique characteristics of different instruments. These factors all increase the complexity and subjectivity of manual annotation. Existing techniques primarily rely on manual annotation and traditional feature extraction methods to process music data. These methods often fail to fully capture the diversity of musical scores, leading to inefficient and inaccurate annotation processes. Furthermore, they struggle to meet the needs of large-scale annotation of folk music scores. Summary of the Invention

[0003] This application provides an automated multi-feature annotation method and equipment for folk music scores, aiming to solve the technical problem that the existing technology mainly relies on manual annotation and traditional feature extraction methods to process music data, which is unable to fully capture the diverse features in the scores, resulting in inefficient and inaccurate annotation process.

[0004] The first aspect disclosed in the present application provides an automated multi-feature tagging method for folk music scores, the method comprising: obtaining a target folk music score corpus, wherein the scores in the target folk music score corpus are all annotated with score categories and performance styles; performing symbol parsing and time-frequency transformation on the target folk music score corpus, and extracting a note sequence matrix and a spectrum feature matrix respectively; constructing an initial feature embedding space through a variational autoencoder based on the note sequence matrix and the spectrum feature matrix; retrieving hierarchical tagging labels, wherein the hierarchical tagging labels include pitch contour, rhythm structure, ornament position and type, and dynamic trend curve; training an initial annotation model based on the hierarchical tagging labels and the initial feature embedding space, wherein the initial annotation model is a Transformer network based on multi-head self-attention; utilizing an elastic weight integration strategy to incrementally fine-tune the initial annotation model to generate a target annotation model; inputting the target folk music score corpus into the target annotation model, and outputting the multi-feature annotation results of the target folk music score.

[0005] The second aspect disclosed in the present application provides an automatic multi-feature tagging device for folk music scores, which is used for the above-mentioned automatic multi-feature tagging method for folk music scores. The device includes: a corpus acquisition module for acquiring a target folk music score corpus, wherein the scores in the target folk music score corpus are all labeled with the score category and performance style; a time-frequency conversion module for performing symbol analysis and time-frequency conversion on the target folk music score corpus, and extracting the note sequence matrix and the spectrum feature matrix respectively; an initial feature acquisition module for constructing an initial feature embedding space through a variational autoencoder based on the note sequence matrix and the spectrum feature matrix; and a label acquisition module for Retrieve hierarchical annotation labels, wherein the hierarchical annotation labels include pitch contour, rhythm structure, ornament position and type, and dynamic trend curve; an initial model training module is used to train an initial annotation model based on the hierarchical annotation labels and the initial feature embedding space, wherein the initial annotation model is a Transformer network based on multi-head self-attention; a target model training module is used to use an elastic weight integration strategy to incrementally fine-tune the initial annotation model to generate a target annotation model; an annotation result acquisition module is used to input the target folk music score corpus into the target annotation model and output the multi-feature annotation results of the target folk music score.

[0006] One or more technical solutions provided in this application have at least the following beneficial effects:

[0007] By establishing a corpus of folk music scores containing annotated musical notation categories and playing styles, the high quality and diversity of the data are ensured, which enables the subsequent annotation model to perform multi-feature annotation based on rich data, thereby improving the accuracy of the model; the music scores are subjected to symbolic analysis and time-frequency transformation of the audio signals, and the note sequence matrix and spectrum feature matrix are extracted respectively. Through symbolic analysis, the pitch, duration and other information of the notes are extracted, and through time-frequency transformation, the spectrum features of the audio signal are extracted. This step provides comprehensive note and spectrum feature data for subsequent model training; through variational autoencoders, the note sequence matrix and spectrum feature matrix are mapped to a low-dimensional latent space, namely the feature embedding space. This feature embedding space can effectively express the comprehensive features of notes and spectra, simplify the dimension of the data, and extract important information in the data, so that the subsequent annotation model can learn features and perform annotations more efficiently; hierarchical annotation labels are extracted, including multiple high-level features such as pitch contour, rhythm structure, ornament type, dynamic trend, etc. These labels help to annotate the music. The symbol sequence and spectral features provide more semantic understanding and enrich the target information of the annotation model. This hierarchical labeling enables the model to distinguish and learn different types of music features, thereby improving the meticulousness and accuracy of the annotation results; by using the Transformer network based on the multi-head self-attention mechanism to train the initial annotation model, it is possible to focus on different parts of the input features at the same time. The self-attention mechanism of the Transformer network can capture the important relationship between different time steps, thereby improving the accuracy of the annotation model when processing complex music score data; through incremental fine-tuning, the model can adapt to newly added folk music score samples, ensuring that the model can maintain memory of existing data and adapt to new samples while continuously receiving new data. Incremental fine-tuning avoids the resource consumption of training the entire model from scratch, allowing the model to be continuously optimized over time, improving the dynamic learning ability of the model; the target annotation model automatically generates accurate multi-feature annotations for folk music scores, greatly improving the efficiency of music score analysis and processing.

[0008] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 A flowchart of the method for automated multi-feature annotation of folk music scores provided in an embodiment of the present application.

[0010] Figure 2 Schematic diagram of the structure of the automatic multi-feature annotation equipment for folk music scores provided in an embodiment of the present application.

[0011] Explanation of the accompanying symbols: corpus acquisition module 10, time-frequency conversion module 20, initial feature acquisition module 30, annotation label acquisition module 40, initial model training module 50, target model training module 60, annotation result acquisition module 70. DETAILED DESCRIPTION

[0012] The embodiments of the present application provide an automated multi-feature labeling method and device for folk music scores, thereby solving the technical problem that the existing technology mainly relies on manual labeling and traditional feature extraction methods to process music data, which cannot fully capture the diverse features in the scores, resulting in an inefficient and inaccurate labeling process.

[0013] After introducing the basic principles of this application, various non-limiting embodiments of this application will be specifically described below in conjunction with the accompanying drawings. It should be understood that the specific embodiments described here are only used to explain this application and are not used to limit this application.

[0014] Example 1, as Figure 1 As shown, the embodiment of the present application provides an automatic multi-feature annotation method for folk music scores, the method comprising:

[0015] A target folk music score corpus is obtained, wherein the scores in the target folk music score corpus are all annotated with the score category and the performance style.

[0016] Collect a certain number of folk music scores and construct a target folk music score corpus. These scores can come from different folk music works or repertoires, covering different styles, instruments, and playing techniques. Each score is annotated with its score type, for example, it can be marked as erhu solo score, flute performance score, or ensemble score. The score type type indicates the performance form or arrangement of the score. Similarly, each score is annotated with the performance style, which involves specific playing techniques, performance methods, and emotional expression, such as fast rhythm and elegant ornamentation.

[0017] The target folk music score corpus is subjected to symbol parsing and time-frequency transformation, and a note sequence matrix and a spectrum feature matrix are extracted respectively.

[0018] Through symbol parsing and time-frequency transformation, the note information and audio features of folk music scores are extracted to prepare for subsequent feature modeling and annotation model training. Specifically, the score symbols are extracted from the target folk music score to form an analyzable symbol set. These symbols include MIDI numbers, time values, part identifiers, types of ornaments, etc. The symbol parsing process uses score parsing software to convert traditional paper or digital music scores (such as XML, MIDI, etc.) into machine-understandable data, and the obtained note data is standardized according to the time value grid and converted into a note event stream. Each note event represents the occurrence of a note and is encoded into a feature vector to generate a note sequence matrix. The note sequence matrix is a two-dimensional matrix that represents the time and pitch information of each note in the folk music score.

[0019] The short-time Fourier transform is used to perform time-frequency transformation on the audio signal in the target folk music score. The short-time Fourier transform divides the audio signal into multiple time domain segments and calculates the spectrum of each segment to generate a two-dimensional frequency-time matrix to represent the spectral characteristics of the signal, that is, the frequency components at each time point. The obtained spectral feature matrix provides the frequency information of the audio for subsequent analysis, which can reflect the pitch, timbre and other details of the performance.

[0020] Based on the note sequence matrix and the spectrum feature matrix, an initial feature embedding space is constructed through a variational autoencoder.

[0021] The note sequence matrix and the spectral feature matrix are concatenated in parallel along the feature dimension to create a joint feature tensor, combining the note and audio spectrum information to create a unified representation. A sliding split operation is performed on the joint feature tensor, dividing the feature tensor into multiple segments according to a preset grid length. The length and splitting method of each segment are determined according to the specific model requirements. Each segment after sliding splitting is normalized to uniformly maintain the data within a numerical range.

[0022] A variational autoencoder is a generative model that can learn a latent space from input data through unsupervised learning. It usually consists of an encoder and a decoder. It takes the sliding segmented data as input and trains it through a variational autoencoder to obtain an initial feature embedding space. This latent space captures the common information of note sequences and spectral features, serving as the basic feature representation for subsequent tasks.

[0023] Retrieve hierarchical annotation tags, wherein the hierarchical annotation tags include pitch contour, rhythm structure, ornament position and type, and dynamic trend curve.

[0024] Retrieve hierarchical annotation tags, including pitch contour, rhythm structure, position and type of ornaments, and dynamic trend curve. The pitch contour describes the changing pattern of notes in a piece of music, which can be expressed as the ups and downs of notes, reflecting the melodic structure of the music; the rhythm structure marks the organizational pattern of notes on the timeline, including the regularity of rhythm and the distribution of rhythmic strength; ornaments are decorative performance techniques for notes, such as glissando and vibrato. This tag marks the specific position and type of ornaments to help annotate the details of the performance; the dynamic trend curve shows the trend of note volume changes. In music performance, dynamic changes can be expressed as crescendo or diminuendo. These tags correspond to the specific features of a part of the score, providing key information for the subsequent annotation model, helping the model learn how to predict these high-level musical attributes through note sequences and spectral features.

[0025] Based on the hierarchical annotation labels and the initial feature embedding space, an initial annotation model is trained, wherein the initial annotation model is a Transformer network based on multi-head self-attention.

[0026] The hierarchical annotation labels are aligned and concatenated with the initial feature embedding space. Specifically, each feature vector is concatenated with its corresponding annotation label into a large input vector. This alignment helps the annotation model focus on both audio features and high-level annotation information. The aligned and concatenated feature vectors are used as the input sequence of the Transformer network. Each input vector contains note and spectrum information as well as high-level information of the hierarchical annotation labels.

[0027] The Transformer network uses a multi-head self-attention mechanism, which can process multiple attention heads in parallel and capture important information in the input data from different perspectives. The self-attention mechanism helps the model identify correlations between different positions in the input sequence. Using the cross-entropy loss function, the training goal is to minimize the gap between the model's predictions and the true labels. Hierarchical annotation labels serve as the target output, guiding the model to learn how to predict the multidimensional characteristics of each note, such as pitch, rhythm, and dynamics, from the input features. During training, the model parameters are continuously updated using music data from the target folk music corpus until the model converges. The model learns the relationship between notes and spectra and makes multi-feature predictions based on the hierarchical annotation labels.

[0028] The initial annotation model is incrementally fine-tuned using an elastic weight integration strategy to generate a target annotation model.

[0029] Acquire new folk music score samples that are not included in the original target folk music score corpus. These new samples can provide more data diversity and help the model improve its adaptability to different scores. Based on the score category and performance style, first calculate the similarity of the new folk music score sample set to evaluate the similarity of each new sample with the original dataset. When the similarity between the new samples and the original dataset is high, the regularization constraints can be relaxed so that the model can be more adaptable to the characteristics of these new samples; when the similarity between the new samples and the original dataset is low, the regularization constraints are strengthened to avoid overfitting the model to these samples that are significantly different from the original data.

[0030] Based on the above judgment, the regularization strength of the model is adjusted, and the initial annotation model is incrementally fine-tuned. During the fine-tuning process, the model will gradually learn how to better adapt to the newly added samples while maintaining the performance on the original samples. After incremental fine-tuning, the target annotation model is obtained, which can effectively process the original data and the newly added data, and provide more accurate and diverse annotation results.

[0031] The target folk music score corpus is input into the target annotation model, and the target folk music score multi-feature annotation results are output.

[0032] The entire target folk music score corpus is input into the trained target annotation model. Based on the input features, the target annotation model uses the multi-head self-attention mechanism of the Transformer network to annotate each score, predict various features such as the pitch, rhythm, ornament type, and dynamic trend of each note, and output the multi-feature annotation results of each score. These annotation results can be directly used for subsequent analysis or performance simulation.

[0033] Furthermore, the target folk music score corpus is subjected to symbol parsing and time-frequency transformation to extract a note sequence matrix and a spectrum feature matrix, respectively, and the method includes:

[0034] The music scores in the target folk music score corpus are symbolically parsed to extract a music score symbol set, wherein the music score symbol set includes a MIDI number, a duration, a voice part identifier, and an ornament sound type; the music score symbol set is converted into a standardized note event stream according to a preset duration grid, and feature encoding is performed to output the note sequence matrix; and time-frequency transformation is performed on the performance audio or MIDI signal in the target folk music score corpus based on short-time Fourier transform to generate the spectrum feature matrix.

[0035] The scores in the target folk music score corpus can be stored in different formats, such as MIDI, MusicXML, etc. Therefore, it is necessary to first perform symbol parsing on these score files. The purpose of the parsing is to extract specific note symbol data from the score. The extracted score symbol set includes MIDI number, duration, part identifier, and grace note type. Among them, each note has a corresponding MIDI number, which represents the pitch of the note, that is, the frequency. Through the MIDI number, the note can be converted into a machine-understandable digital representation; the duration of each note, that is, the duration, refers to the length of time the note is played in the score. Common durations include whole notes, half notes, quarter notes, etc.; in complex scores, there are usually multiple parts, and part identifiers can help distinguish different parts and clarify the performance information of each part; grace notes are modifications of the main notes, such as glissando, vibrato, etc., and the grace note types include appoggiatura before the note, glissando, the type and position of grace notes, etc.

[0036] The preset duration grid is a preset rule used to divide the note events on the timeline into fixed time units, namely grids. Common duration grids include 1 / 4, 1 / 8, 1 / 16, etc., which represent the resolution of the time unit. For each note in the note symbol set, it is mapped to the corresponding time grid according to its duration. Different note durations will occupy different durations in the grid. Each note is converted into a standardized note event stream according to the duration grid. In the note event stream, each note is represented as an event, including its MIDI number, the time point of occurrence (determined by the duration grid) and the duration. This conversion serializes the notes in the score in time, making the playing order and time relationship of the notes clearer.

[0037] Each note event is encoded as a feature vector, including the note's pitch (MIDI number), duration, voice ID, and other information. Feature encoding methods can include one-hot encoding, integer encoding, or other numerical representations. The features of each note event will be converted into a numerical matrix, where each time step corresponds to a note event.

[0038] The feature encoding results of all note event streams are organized into a note sequence matrix. The rows of this matrix represent different time steps, the columns represent different features, and each element in the matrix represents the value of a certain feature at that time step.

[0039] Perform time-frequency analysis on the performance audio or MIDI signals in the target folk music score corpus. If it is a MIDI signal, convert it into an audio signal; if it is an existing audio file, such as WAV or MP3 format, process it directly.

[0040] The short-time Fourier transform (SFT) is a commonly used time-frequency analysis method that segments an audio signal into multiple short time segments. The Fourier transform (SFT) is then applied to each segment to extract its spectral information. The SFT converts the time-domain signal into a frequency-domain representation, revealing the frequency components of the audio over different time periods. For each time segment, the SFT outputs a complex spectral representation containing amplitude and phase information. After the time-frequency transform, the spectral information of each audio segment forms a spectral feature matrix, where rows represent different frequencies and columns represent different time periods. This matrix reflects the frequency distribution of the audio signal over different time periods, including frequency-domain features such as pitch, timbre, and amplitude.

[0041] Furthermore, the method of constructing an initial feature embedding space based on the note sequence matrix and the spectrum feature matrix through a variational autoencoder includes:

[0042] The note sequence matrix and the spectrum feature matrix are spliced in parallel in the feature dimension to obtain a joint feature tensor; the joint feature tensor is slidingly segmented according to a preset grid length, and the sliding segmentation result is normalized and input into the variational autoencoder. After the training is completed, the initial feature embedding space is obtained.

[0043] The note sequence matrix and the spectrum feature matrix are concatenated in the feature dimension to obtain a new joint feature tensor. The concatenation operation connects the two matrices in the column dimension, that is, the feature dimension, to form a new matrix. This tensor contains all the information of the note sequence and spectrum features, providing a more comprehensive feature representation for subsequent training.

[0044] Set a fixed time window, that is, the grid length. For example, each segment contains 10 time steps or other unit time length. This length will determine how to split the joint feature tensor. Use the sliding window technology to split the joint feature tensor. The sliding window will start from the starting position of the tensor and split it into multiple segments according to the set time window length. Each slice contains a fixed time period, and the feature information within the time period will be used for learning.

[0045] Each small segment obtained by sliding segmentation is normalized. Normalization is to make the value of each slice distributed in a similar range, so as to avoid certain features having a disproportionate impact on the model during training due to different scales. For example, the minimum and maximum normalization method is used to scale the data to the range of [0,1].

[0046] The normalized sliding segmentation results are input into a variational autoencoder (VAE). This unsupervised learning model aims to learn a latent space representation of the input data. It consists of an encoder and a decoder. The encoder maps the input features to the latent space, generating a distribution of latent variables; the decoder samples from the latent space and attempts to reconstruct the input data. The goal of the variational autoencoder is to learn the latent representation of the data by minimizing the reconstruction error and KL divergence. Backpropagation is used to optimize the parameters to obtain the initial feature embedding space. This space can effectively represent the low-dimensional representation of the input data. The latent space captures the combined information of the note sequence and spectral features for subsequent tasks.

[0047] Furthermore, the method of training an initial annotation model based on the hierarchical annotation labels and the initial feature embedding space includes:

[0048] The hierarchical annotation labels and the initial feature embedding space are aligned and spliced to obtain a feature input vector; temporal position encoding is superimposed on the feature input vector, and the hierarchical annotation labels are embedded in the output of each layer of the Transformer network. The target folk music score corpus is iteratively jointly trained using cross-entropy loss until the model converges, thereby obtaining the initial annotation model.

[0049] The hierarchical annotation labels include musical features such as pitch contour, rhythmic structure, position and type of ornaments, and dynamic trend curve. Each annotation label corresponds to a specific musical attribute and has high-level semantic information. These labels are usually discrete labels, representing specific behaviors or performances in musical works, such as pitch change trends, rhythm organization, dynamic changes, etc.; the initial feature embedding space is obtained through a variational autoencoder, which contains a low-dimensional latent representation of the note sequence matrix and the spectral feature matrix. This space provides a more concise and expressive feature representation that can capture the relationship between notes and spectra.

[0050] The hierarchical annotation labels are aligned with the latent representation in the initial feature embedding space. The alignment method can be time-step alignment, that is, the label of each time step is aligned with the corresponding note and spectrum features to ensure the temporal consistency of the label and the input feature. After alignment, each feature vector is spliced together to form a new feature input vector. The spliced vector contains the features of each time step and the corresponding hierarchical label information.

[0051] The temporal position encoding is superimposed on the feature input vector. Position encoding is a mechanism used in the Transformer network to maintain the position relationship of elements in the sequence. Because the Transformer does not have a recursive structure, position encoding is needed to explicitly add the sequential information of the time step. The position encoding can be a fixed value generated by the sine and cosine functions, or a trainable parameter learned through training. Superimposing the position encoding on each feature input vector enables the model to perceive the sequential information in the input data.

[0052] Hierarchical annotation labels are embedded in the output of each Transformer layer. This is to incorporate the high-level information of the labels into each attention head of the model, helping the model capture deeper musical features and relationships. By embedding hierarchical annotation labels, the model can be guided to better understand the relationship between note and spectral information and the high-level structure of music.

[0053] The output of the Transformer model is compared with the target hierarchical annotated labels. The difference between the model's predictions and the true labels is measured by calculating the cross-entropy loss. Cross-entropy loss is a commonly used loss function in classification tasks and can effectively optimize the model. During training, the note sequences and spectral features from the target folk music corpus are fed into the model along with the true annotated labels. The model parameters are optimized through backpropagation and gradient descent until the loss is minimized and the model converges. This means that the loss function no longer changes significantly or the preset number of iterations is reached. At this point, the model has learned to make effective predictions from the note sequences, spectral features, and hierarchical labels.

[0054] Furthermore, the method of iteratively joint training the target folk music score corpus by cross entropy loss includes:

[0055] The hierarchical annotation labels are divided into primary and secondary features to obtain primary feature labels and secondary feature labels; based on the primary feature labels and secondary feature labels, the output of the Transformer network is respectively sent to the primary feature prediction classifier and the secondary feature prediction classifier; and the total loss function is called, wherein the total loss function is as follows:

[0056] L=λ main L main +λ aux L aux ;

[0057] Among them, L represents the total loss function, λ main ,λ aux The hyperparameters that characterize the main feature labels and auxiliary feature labels, L main Characterizes the cross entropy loss of the main feature prediction classifier, L auxCharacterizing the cross entropy loss of the auxiliary feature prediction classifier; performing iterative training based on the total loss function until the model converges to obtain the initial annotation model.

[0058] The hierarchical annotation labels are divided into primary and secondary features, so that the model can focus on the primary and secondary features respectively, thereby making better predictions. Among them, the primary feature label is the core prediction target of the model, including the most critical features in music, such as pitch contour, rhythm structure and other main features, which are the main direction of the model annotation; the auxiliary feature label is additional information related to the primary feature, such as the position and type of ornaments, dynamic trends, etc. Although it has an impact on the overall performance of the notes, it is not a core feature.

[0059] The output of the Transformer network is the representation learned by the model from the input features. It contains a combination of information about note sequences, spectral features, and hierarchical annotation labels. Based on the primary feature labels, the output of the Transformer network is fed into a primary feature prediction classifier. This classifier is specifically designed to predict primary feature labels, such as pitch contour and rhythmic structure. Based on the secondary feature labels, the output of the Transformer network is fed into an auxiliary feature prediction classifier. This classifier is specifically designed to predict auxiliary feature labels, such as the type of ornamentation and dynamic trends.

[0060] Retrieve the total loss function as follows:

[0061] L=λ main L main +λ aux L aux ;

[0062] Specifically, the total loss function consists of two parts, one is the loss of the main feature L main , used to measure the error of the model in predicting the main feature label, and the other is the loss L of the auxiliary feature aux , which is used to measure the error of the model in predicting the auxiliary feature labels. Through the hyperparameter λ of the main feature label and the auxiliary feature label main ,λ aux , can balance the influence of main features and auxiliary features in training, and adjust λ according to the priority of the task and the importance of the feature. main ,λ aux The value of ensures that the model can fully learn both the main features and the auxiliary features.

[0063] The gradient is calculated based on the total loss function, and the model parameters are adjusted through the back-propagation algorithm. In each round of training, the model output is compared with the true label, and the prediction ability of the model is optimized by calculating the loss function. The training process will continue until the total loss function reaches a minimum value or no longer changes significantly in multiple training rounds, indicating that the model has converged, that is, the model parameters have reached the optimal state on the given training data. At this time, the initial annotation model is output, which can predict a variety of musical features such as pitch and rhythm based on the input folk music score.

[0064] Furthermore, the method of incrementally fine-tuning the initial annotation model further includes:

[0065] Acquire a newly added folk music score sample set; perform similarity calculation on the newly added folk music score sample set based on the score category and the performance style to obtain a sample similarity set; judge the sample similarity set based on a similarity threshold, and perform incremental fine-tuning of the regularization strength according to the judgment result.

[0066] We acquire new sets of folk music score samples from diverse performers, performance styles, and even audio recordings. These samples can be new pieces or new interpretations of existing pieces. These samples must be formatted in the same format as the existing target folk music score corpus to facilitate seamless integration and training. These new samples help fine-tune the model on diverse data, improving its adaptability to various styles and variations.

[0067] Similarity metrics, such as cosine similarity and Euclidean distance, are used to measure the similarity between the newly added samples and the existing target folk music score samples in terms of musical pattern categories and performance styles. The similarity value indicates the degree of proximity between the newly added samples and the existing samples in terms of musical pattern categories and performance styles. For musical pattern categories, the similarity is calculated by comparing the labels of musical pattern categories, such as erhu solo and flute solo. For performance styles, the similarity between samples is calculated by comparing the characteristics of the performance styles, such as rhythm, dynamics, and ornaments. Similarity calculations can be performed using audio features (such as MFCC and spectrogram features) or annotated hierarchical labels.

[0068] Based on the similarity calculation of the spectrum category and playing style, a sample similarity set is generated, which contains the similarity information between all new samples and existing samples, helping the model to determine the degree of correlation between new samples and existing data.

[0069] Set a predefined similarity threshold. This value is usually determined through experiments and can be a floating value. It is used to determine whether the similarity between the new sample and the existing sample is high enough. When the similarity between the new sample and the existing sample is higher than the threshold, it means that the new sample is highly similar to the existing data. For new samples with high similarity, the model can use a smaller regularization strength during fine-tuning, so that the model can adjust the weight more flexibly and fully learn the characteristics of the new sample. When the similarity is lower than the threshold, it means that the characteristics of the new sample and the existing data are very different. For new samples with low similarity, the regularization constraints can be strengthened to prevent the model from over-adjusting parameters and avoid overfitting these samples with large differences. Adjust the regularization strategy based on the judgment results, and then use the incremental fine-tuning method to update the existing model. Update the model parameters through backpropagation until the model converges and can adapt to the new samples.

[0070] Furthermore, the method of judging the sample similarity set based on the similarity threshold and adjusting the regularization strength by incremental fine-tuning according to the judgment result includes:

[0071] Extracting a first sample similarity and a corresponding first newly added folk music score sample; when the first sample similarity is greater than or equal to the similarity threshold, relaxing the regularization constraint on the first newly added folk music score sample; when the first sample similarity is less than the similarity threshold, strengthening the regularization constraint on the first newly added folk music score sample.

[0072] A sample is randomly selected from the newly added folk music score sample set as the first newly added folk music score sample and the current analysis object. Using information such as the score category and performance style, the similarity between the newly added sample and the samples in the existing target folk music score corpus is calculated to obtain the first sample similarity, which is used to evaluate the closeness of the sample to the existing data set.

[0073] If the similarity of the first sample is greater than or equal to the preset similarity threshold, it means that the new sample has a high similarity with the existing data, and the model can learn the information of the sample more easily. When the similarity is high, the regularization constraint is relaxed, allowing the model to adjust parameters more to adapt to the new sample. This helps the model learn the characteristics of the new sample more flexibly and avoid excessive regularization restrictions. This can be achieved by reducing the L2 regularization term or modifying the regularization strength, thereby allowing the model to adjust more weights.

[0074] If the similarity of the first sample is less than the preset similarity threshold, it means that there are significant differences between the new samples and the existing data, and the model may overfit due to learning these samples. When the similarity is low, the overfitting of the model to the new samples can be reduced by strengthening the regularization constraints. Strengthening regularization helps limit the excessive adjustment of the model to the features of the new samples during the learning process, thereby improving the generalization ability of the model. This can be achieved by increasing the L2 regularization term, using a higher regularization strength, or limiting the amplitude of parameter updates during the optimization process.

[0075] In summary, the automated multi-feature annotation method for folk music scores provided in the embodiments of the present application has the following technical effects:

[0076] By establishing a corpus of folk music scores containing annotated musical notation categories and playing styles, the high quality and diversity of the data are ensured, which enables the subsequent annotation model to perform multi-feature annotation based on rich data, thereby improving the accuracy of the model; the music scores are subjected to symbolic analysis and time-frequency transformation of the audio signals, and the note sequence matrix and spectrum feature matrix are extracted respectively. Through symbolic analysis, the pitch, duration and other information of the notes are extracted, and through time-frequency transformation, the spectrum features of the audio signal are extracted. This step provides comprehensive note and spectrum feature data for subsequent model training; through variational autoencoders, the note sequence matrix and spectrum feature matrix are mapped to a low-dimensional latent space, namely the feature embedding space. This feature embedding space can effectively express the comprehensive features of notes and spectra, simplify the dimension of the data, and extract important information in the data, so that the subsequent annotation model can learn features and perform annotations more efficiently; hierarchical annotation labels are extracted, including multiple high-level features such as pitch contour, rhythm structure, ornament type, dynamic trend, etc. These labels help to annotate the music. The symbol sequence and spectral features provide more semantic understanding and enrich the target information of the annotation model. This hierarchical labeling enables the model to distinguish and learn different types of music features, thereby improving the meticulousness and accuracy of the annotation results; by using the Transformer network based on the multi-head self-attention mechanism to train the initial annotation model, it is possible to focus on different parts of the input features at the same time. The self-attention mechanism of the Transformer network can capture the important relationship between different time steps, thereby improving the accuracy of the annotation model when processing complex music score data; through incremental fine-tuning, the model can adapt to newly added folk music score samples, ensuring that the model can maintain memory of existing data and adapt to new samples while continuously receiving new data. Incremental fine-tuning avoids the resource consumption of training the entire model from scratch, allowing the model to be continuously optimized over time, improving the dynamic learning ability of the model; the target annotation model automatically generates accurate multi-feature annotations for folk music scores, greatly improving the efficiency of music score analysis and processing.

[0077] Example 2, based on the same inventive concept as the method for automatically labeling folk music scores in the above embodiment, Figure 2 As shown, the embodiment of the present application provides an automatic multi-feature annotation device for folk music scores, the device comprising:

[0078] The corpus acquisition module 10 is used to acquire the target folk music score corpus, wherein the scores in the target folk music score corpus are all marked with the score category and performance style; the time-frequency conversion module 20 is used to perform symbol analysis and time-frequency conversion on the target folk music score corpus, and extract the note sequence matrix and the spectrum feature matrix respectively; the initial feature acquisition module 30 is used to construct the initial feature embedding space through the variational autoencoder based on the note sequence matrix and the spectrum feature matrix; the annotation label acquisition module 40 is used to call the hierarchical annotation label, wherein the hierarchical annotation label includes the pitch contour, the rhythm The method comprises the following steps: an initial model training module 50 for training an initial annotation model based on the hierarchical annotation labels and the initial feature embedding space, wherein the initial annotation model is a Transformer network based on multi-head self-attention; a target model training module 60 for performing incremental fine-tuning on the initial annotation model using an elastic weight integration strategy to generate a target annotation model; an annotation result acquisition module 70 for inputting the target folk music score corpus into the target annotation model and outputting the multi-feature annotation results of the target folk music score.

[0079] Furthermore, the time-frequency conversion module 20 is configured to perform the following steps:

[0080] The music scores in the target folk music score corpus are symbolically parsed to extract a music score symbol set, wherein the music score symbol set includes a MIDI number, a duration, a voice part identifier, and an ornament sound type; the music score symbol set is converted into a standardized note event stream according to a preset duration grid, and feature encoding is performed to output the note sequence matrix; and time-frequency transformation is performed on the performance audio or MIDI signal in the target folk music score corpus based on short-time Fourier transform to generate the spectrum feature matrix.

[0081] Furthermore, the initial feature acquisition module 30 is configured to perform the following steps:

[0082] The note sequence matrix and the spectrum feature matrix are spliced in parallel in the feature dimension to obtain a joint feature tensor; the joint feature tensor is slidingly segmented according to a preset grid length, and the sliding segmentation result is normalized and input into the variational autoencoder. After the training is completed, the initial feature embedding space is obtained.

[0083] Furthermore, the initial model training module 50 is used to perform the following steps:

[0084] The hierarchical annotation labels and the initial feature embedding space are aligned and spliced to obtain a feature input vector; temporal position encoding is superimposed on the feature input vector, and the hierarchical annotation labels are embedded in the output of each layer of the Transformer network. The target folk music score corpus is iteratively jointly trained using cross-entropy loss until the model converges, thereby obtaining the initial annotation model.

[0085] Furthermore, the initial model training module 50 is used to perform the following steps:

[0086] The hierarchical annotation labels are divided into primary and secondary features to obtain primary feature labels and secondary feature labels; based on the primary feature labels and secondary feature labels, the output of the Transformer network is respectively sent to the primary feature prediction classifier and the secondary feature prediction classifier; and the total loss function is called, wherein the total loss function is as follows:

[0087] L=λ main L main +λ aux L aux ;

[0088] Among them, L represents the total loss function, λ main ,λ aux The hyperparameters that characterize the main feature labels and auxiliary feature labels, L main Characterizes the cross entropy loss of the main feature prediction classifier, L aux Characterizing the cross entropy loss of the auxiliary feature prediction classifier; performing iterative training based on the total loss function until the model converges to obtain the initial annotation model.

[0089] Furthermore, the target model training module 60 is configured to perform the following steps:

[0090] Acquire a newly added folk music score sample set; perform similarity calculation on the newly added folk music score sample set based on the score category and the performance style to obtain a sample similarity set; judge the sample similarity set based on a similarity threshold, and perform incremental fine-tuning of the regularization strength according to the judgment result.

[0091] Furthermore, the target model training module 60 is configured to perform the following steps:

[0092] Extracting a first sample similarity and a corresponding first newly added folk music score sample; when the first sample similarity is greater than or equal to the similarity threshold, relaxing the regularization constraint on the first newly added folk music score sample; when the first sample similarity is less than the similarity threshold, strengthening the regularization constraint on the first newly added folk music score sample.

[0093] Through the detailed description of the method for automatic multi-feature annotation of folk music scores in the foregoing specification, those skilled in the art can clearly understand the automatic multi-feature annotation device for folk music scores in this embodiment. Since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the description of the method part.

[0094] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present application. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application is not limited to the embodiments shown herein, but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An automated multi-feature annotation method for folk music scores, characterized in that: The method comprises: Obtaining a target folk music score corpus, wherein the scores in the target folk music score corpus are all annotated with the score category and performance style; Performing symbol parsing and time-frequency transformation on the target folk music score corpus to extract a note sequence matrix and a spectrum feature matrix respectively; Based on the note sequence matrix and the spectrum feature matrix, constructing an initial feature embedding space through a variational autoencoder; Retrieving hierarchical annotation tags, wherein the hierarchical annotation tags include pitch contour, rhythm structure, ornament position and type, and dynamic trend curve; Based on the hierarchical annotation labels and the initial feature embedding space, training an initial annotation model, wherein the initial annotation model is a Transformer network based on multi-head self-attention; Using an elastic weight integration strategy, incrementally fine-tuning the initial annotation model to generate a target annotation model; The target folk music score corpus is input into the target annotation model, and the target folk music score multi-feature annotation results are output.

2. The automatic multi-feature annotation method for folk music scores according to claim 1, characterized in that: The method of performing symbol parsing and time-frequency transformation on the target folk music score corpus to extract a note sequence matrix and a spectrum feature matrix respectively includes: Performing symbol parsing on the music scores in the target folk music score corpus to extract a music score symbol set, wherein the music score symbol set includes a MIDI number, a duration, a voice part identifier, and an ornament note type; Converting the music score symbol set into a standardized note event stream according to a preset time value grid, performing feature encoding, and outputting the note sequence matrix; The performance audio or MIDI signal in the target folk music score corpus is subjected to time-frequency transformation based on short-time Fourier transform to generate the spectrum feature matrix.

3. The automatic multi-feature annotation method for folk music scores according to claim 2, characterized in that: The method of constructing an initial feature embedding space based on the note sequence matrix and the spectrum feature matrix through a variational autoencoder includes: Concatenate the note sequence matrix and the spectrum feature matrix in parallel in the feature dimension to obtain a joint feature tensor; The joint feature tensor is subjected to sliding segmentation according to a preset grid length, and the sliding segmentation result is normalized and then input into the variational autoencoder, and the initial feature embedding space is obtained after the training is completed.

4. The automatic multi-feature annotation method for folk music scores according to claim 1, wherein: The method of training an initial annotation model based on the hierarchical annotation labels and the initial feature embedding space includes: Aligning and splicing the hierarchical annotation labels and the initial feature embedding space to obtain a feature input vector; The temporal position encoding is superimposed on the feature input vector, and the hierarchical annotation label is embedded in the output of each layer of the Transformer network. The target folk music score corpus is iteratively jointly trained through cross-entropy loss until the model converges, thereby obtaining the initial annotation model.

5. The automatic multi-feature annotation method for folk music scores according to claim 4, characterized in that: The method of iteratively joint training the target folk music score corpus by cross entropy loss includes: Dividing the hierarchical annotation labels into primary and secondary features to obtain primary feature labels and secondary feature labels; Based on the main feature label and the auxiliary feature label, the output of the Transformer network is respectively sent to the main feature prediction classifier and the auxiliary feature prediction classifier; The total loss function is retrieved, where the total loss function is as follows: L=λ main L main +λ aux L aux ; Among them, L represents the total loss function, λ main ,λ aux The hyperparameters that characterize the main feature labels and auxiliary feature labels, L main Characterizes the cross entropy loss of the main feature prediction classifier, L aux Cross entropy loss that characterizes the auxiliary feature prediction classifier; Iterative training is performed based on the total loss function until the model converges to obtain the initial annotation model.

6. The automatic multi-feature annotation method for folk music scores according to claim 1, wherein: The incremental fine-tuning of the initial annotation model further comprises: Get a new set of folk music score samples; Calculating similarity of the newly added folk music score sample set based on the score category and the performance style to obtain a sample similarity set; The sample similarity set is judged based on a similarity threshold, and regularization strength adjustment is performed by incremental fine-tuning according to the judgment result.

7. The automatic multi-feature annotation method for folk music scores according to claim 6, characterized in that: The method of judging the sample similarity set based on the similarity threshold and adjusting the regularization strength by incremental fine-tuning according to the judgment result includes: Extracting the first sample similarity and the corresponding first newly added folk music score sample; When the first sample similarity is greater than or equal to the similarity threshold, relaxing the regularization constraint on the first newly added folk music score sample; When the first sample similarity is less than the similarity threshold, regularization constraints are strengthened on the first newly added folk music score sample.

8. Automatic multi-feature annotation equipment for folk music scores, characterized by: The device for implementing the automatic multi-feature annotation method for folk music scores according to any one of claims 1 to 7 comprises: A corpus acquisition module is used to acquire a target folk music score corpus, wherein the scores in the target folk music score corpus are all annotated with the score category and performance style; A time-frequency conversion module is used to perform symbol parsing and time-frequency conversion on the target folk music score corpus, and extract a note sequence matrix and a spectrum feature matrix respectively; An initial feature acquisition module, configured to construct an initial feature embedding space through a variational autoencoder based on the note sequence matrix and the spectrum feature matrix; A label acquisition module is used to retrieve hierarchical labeling tags, wherein the hierarchical labeling tags include pitch contour, rhythm structure, ornament position and type, and dynamic trend curve; An initial model training module, configured to train an initial annotation model based on the hierarchical annotation labels and the initial feature embedding space, wherein the initial annotation model is a Transformer network based on multi-head self-attention; a target model training module, configured to incrementally fine-tune the initial annotation model using an elastic weight integration strategy to generate a target annotation model; The annotation result acquisition module is used to input the target folk music score corpus into the target annotation model and output the multi-feature annotation results of the target folk music score.

Images