Music stave sentiment classification method and system based on multi-level distillation

Abstract

The invention discloses a music stave sentiment classification method and system based on multi-level distillation, and belongs to the technical field of music information processing. The method comprises the steps of firstly collecting music stave data and converting the data into stave data vectors, then performing feature extraction by using a long short-term memory network, then constructing a teacher network and a student network for knowledge distillation, and realizing multi-level knowledge transmission through temperature scaling, KL divergence loss and mask feature distillation. And finally, training a lightweight classification model to complete sentiment classification. The knowledge distillation technology is creatively applied to staff sentiment classification, the classification accuracy is effectively improved through an online multi-level distillation mode, and the technical problems that a traditional method lacks semantic information and a self-supervised model is not suitable for sentiment tasks are solved. The method has the main advantages of high classification precision, light model weight, capability of effectively capturing music emotion features and the like.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of music information processing, and relates to a music staff emotion classification method and system based on multi-level distillation. BACKGROUND

[0002] Music, as a universal language of human emotion, its inheritance and expression is highly dependent on the notation system. Among many notation methods, the staff has become a global "text" of music due to its scientificity and universality, building a bridge of emotional communication between composers and performers.

[0003] The staff is a way to record music, composed of five parallel horizontal lines, the interval between these lines is called "interval", the uppermost line is the fifth line, and the lowermost line is called the first line, the interval between adjacent two lines is called the first interval, the second interval, the third interval and the fourth interval. In the staff, the note is composed of three parts: the head, the stem and the tail. The note is a symbol representing the length of the sound or the time value, and the rest symbol represents the duration of the sound pause or interval. When recording music, a set of marking rules needs to be followed. Correct understanding and application of the rules can make the score clear and understandable, accurately convey the emotion of music, and promote the exchange between musicians.

[0004] Currently, extracting emotion representation from the staff is a key step in various emotional tasks such as music emotion recognition and emotion analysis. Traditional methods usually use filter banks or mel-frequency cepstral coefficients as music features, but these features lack semantic information, resulting in limited performance in emotional tasks. The current mainstream method extracts features by pre-training a model based on music self-supervised learning, which improves performance in some tasks, but these self-supervised learning models are not completely suitable for emotional tasks. SUMMARY

[0005] In order to improve the accuracy of music staff emotion classification, the present application provides a music staff emotion classification method and system based on multi-level distillation.

[0006] The present application is realized by the following technical scheme: a music staff emotion classification method based on multi-level distillation, comprising the following steps: step 1, collecting music staff data, and constructing a training data set containing multiple emotion types; Step 2, converting the staff into a staff data vector, including embedding representation and splicing processing of pitch and dynamics; Step 3, using a long short-term memory network to extract features from the staff data vector to obtain staff features; Step 4, constructing a teacher network and a student network for knowledge distillation, and realizing knowledge transfer through temperature scaling and KL divergence loss; Step 5, fix the student network's Transformer backbone network parameters, and increase the ResNet model for lightweight classification training.

[0007] Specifically, the step 2 specifically comprises: The pitch data and the dynamics data are embedded and transformed by using a pitch embedding module and a dynamics embedding module respectively, so as to obtain a pitch embedding vector and a dynamics embedding vector respectively. The pitch and dynamics of each track in each sampling period are collected. The pitch data is obtained by the pitch and the pitch embedding vector, and the dynamics data is obtained by the dynamics and the dynamics embedding vector, and after the pitch data and the dynamics data are spliced, a stave data vector is obtained through a linear fully connected layer.

[0008] Specifically, the long short-term memory network in the step 3 comprises: A forgetting gate is used to determine the forgetting degree of the output information of the previous position; An input gate is used to update the memory cell state at the current time; An output gate is used to control the output value at the current time; A memory cell is used to store long-term dependent information.

[0009] Specifically, the knowledge distillation in the step 4 comprises: The outputs of the teacher network and the student network are softened by using a temperature parameter; The difference between the output distributions of the teacher network and the student network is measured by using a KL divergence loss; A mask feature distillation mechanism is designed, and the feature dimension is adapted by using bilinear interpolation.

[0010] Specifically, the teacher network comprises a long short-term memory network, a multi-level Transformer backbone network and a softmax activation function in sequence, and the student network comprises a long short-term memory network, a mask module, a multi-level Transformer backbone network and a softmax activation function in sequence.

[0011] Specifically, the total loss of the knowledge distillation comprises a KL divergence loss and a mask feature loss.

[0012] The application also provides a music stave emotion classification system, comprising: A data acquisition module is used to acquire and preprocess music stave data; A vector conversion module is used to convert the stave into a stave data vector; A feature extraction module uses a long short-term memory network to extract stave features; A knowledge distillation module is used to realize the knowledge transfer from the teacher network to the student network. The classification module is used to complete the final sentiment classification task.

[0013] Furthermore, the knowledge distillation module includes a teacher network and a student network. The teacher network consists of a long short-term memory network, a multi-level Transformer backbone network, and a softmax activation function. The student network adds a masking module to the teacher network.

[0014] The present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements a music staff emotion classification method based on multi-level distillation.

[0015] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement a music staff emotion classification method based on multi-level distillation.

[0016] Through specially designed embedding layers such as pitch embedding and velocity embedding, the structure information of the musical staff is effectively encoded, enabling the model to understand the deep relationship between musical elements such as notes and rhythms and emotional expression, and providing richer feature representations for music emotion analysis.

[0017] This invention innovatively combines LSTM temporal feature extraction with a Transformer backbone network, preserving the long-term dependencies of music sequences while capturing global features through the Transformer's self-attention mechanism. The mask feature distillation mechanism further enhances the collaborative learning of local details and the overall structure.

[0018] This invention employs an online, multi-level distillation method for self-supervised pre-training on sentiment data. Considering that both overall information and local details can convey sentiment, a pre-training strategy combining multiple loss mechanisms is used, providing a robust representational foundation for subsequent self-supervised guided learning and effectively optimizing the online distillation process. Attached Figure Description

[0019] Figure 1 This is a schematic diagram of knowledge distillation according to the present invention.

[0020] Figure 2 Comparison chart of recognition rate across training iterations. Detailed Implementation

[0021] The present invention will be further explained in detail below with reference to the embodiments.

[0022] Reference Figure 1 The first embodiment of the present invention provides a music staff notation emotion classification method based on multi-level distillation, the steps of which are as follows: Step 1: Collect musical staff data.

[0023] Through the study of music, musicologists have categorized music into eight types of emotions, as shown in Table 1. Each column represents one type of emotion, and each emotion has many words to represent it. The words in each group have very similar meanings.

[0024] Table 1

[0025] The music staff notation dataset used in this embodiment includes a self-built dataset and an open-source dataset. The self-built dataset is based on the eight different emotion types in Table 1, and for each emotion type, it collects ten different styles of staff notation. The complete music staff notation dataset is represented as follows: ,in This represents the number of musical staff samples, where n is the index of the musical staff sample. This is the nth musical staff.

[0026] Step 2: Convert the musical staff into a musical staff data vector. Before extracting features from the musical staff, it needs to be represented as data. In this embodiment, an embedding module is used to map different types of vector data on the musical staff to different embedding vector spaces and then concatenate them. Additionally, embeddings related to musical structure, including pitch embeddings, are added to the data. and force embedding After performing embedding transformations and linear mappings on the word vectors of pitch and dynamics respectively, this invention adds two types of information related to musical position—pitch and dynamics—to each sampling point in the data sequence. This invention uses 1 / 4 of a beat as a unit sampling period, simultaneously acquiring the first [number] [unit] of the musical staff. Sampling period Pitch of the track and strength Then the first Sampling period Track pitch data Represented as: Strength data Represented as: ,in This indicates a concatenation operation, used to join two vectors.

[0027] Then the first Sampling period The orbital data is represented as Similarly, the pitch and velocity data are concatenated, and then processed by a fully connected linear layer to obtain the staff data vector. , This represents a linear fully connected layer.

[0028] Step 3: Use a Long Short-Term Memory (LSTM) network to extract features from the staff data vector to obtain the staff features; Long Short-Term Memory (LSTM) networks are a variant of recurrent neural networks. LSTM networks consist of many repeating units called memory blocks. Each memory block contains three gates and one memory unit. The three gates are the forget gate, the input gate, and the output gate.

[0029] Forget Gate: The first element in the LSTM's internal structure is the forget gate, which determines which parts of the output information from the previous position should be forgotten. The input to this layer is the output from the previous position. and the input of the current position After making connections, performing a linear transformation, and finally using the sigmoid activation function to select which information is forgotten, the calculation formula is as follows: ; in, Indicates the output of the forget gate. This represents the musical staff data vector at time t. Indicates the forget gate bias term. This represents the weight parameter matrix of the forget gate. This represents the sigmoid activation function. When the sigmoid function is closer to 0, it indicates forgetting; when it is closer to 1, it indicates that memory is needed.

[0030] Input gate: The second part of the LSTM internal structure is the input gate. It mainly consists of two parts; the input is also the output of the previous position. and the input of the current position First, a connection is established, followed by a linear transformation. The first part updates the information, and the second part determines which parts need to be updated. First, a sigmoid activation function layer determines which values ​​need to be updated, then... The layer updates the input information. The calculation formula is: ; ; in, Indicates the output of the input gate. This represents the updated weight parameter matrix of the input gate. This represents the bias term of the input gate. This indicates updating the output at the previous position. state, express The layer's weight parameter matrix, for Layer bias terms; Output gate: The input to the output gate is obtained by processing the model input. The output calculation formula for the output gate is: ,in, This represents the updated weight parameter matrix of the output gate. This represents the bias term of the output gate.

[0031] Model output: Current state of the memory unit The calculation uses the forget gate to output. and the output of the input gate The state of the memory unit at the previous moment Used to calculate the current state of the memory cell Output the forget gate respectively and the state of the memory unit at the previous moment Element-wise multiplication, the output of the input gate and Element-wise multiplication yields the current state of the memory cell. The process is as follows: ,in This indicates element-wise multiplication, and the output of the memory unit is in the form of a musical staff. , It will be further passed to the upper layers of the Long Short-Term Memory network or used as the final prediction result.

[0032] Step 4: Construct teacher and student networks for knowledge distillation; The teacher network is a pre-trained complete model, a powerful model trained on a large amount of data. The student network is a relatively lightweight small model. By adjusting the parameters or network structure of the student network, it can better capture the information features learned by the teacher network, thereby improving the generalization ability of the target model. In this invention, the student network masks the spectral features. The teacher network consists of a Long Short-Term Memory (LSTM) network, a multi-layered Transformer backbone network, and a softmax activation function. The student network consists of an LSTM network, a mask module, a multi-layered Transformer backbone network, and a softmax activation function. Figure 1 As shown in (a). The mask module (mask) requires the mask region to be generated randomly. In this invention, the mask region ratio is 35%. The student network updates parameters through backpropagation, while the teacher network uses exponential moving average (EMA). The parameter update uses a multi-level distillation loss design. The distillation process is optimized through multiple levels of loss functions, including the use of temperature softening and KL divergence loss. The knowledge of the teacher network is transferred to the student network to train the student network to represent the internal emotional features of the staff.

[0033] To effectively guide the learning process of students' networks, a multi-level distillation loss function was designed. First, for the k-th output data of the teacher network... and student network output data Temperature scaling was applied to the output data of the teacher network and the student network respectively, and the results are as follows: ; ; in, Let be the softening probability distribution output by the teacher network in the kth iteration. Let be the softening probability distribution output by the student network in the kth iteration. For temperature parameters, Output data for the k-th iteration of the teacher network. Output data for the j-th iteration of the teacher network. Output data for the k-th iteration of the student network. This outputs data for the j-th iteration of the student network. At high temperatures, the softening probability distribution is smoother, and the knowledge from the teacher network is transferred to the student network in this way. A higher temperature coefficient is used in the early stages of training. As training progresses, the temperature coefficient is gradually reduced to approximately 1. This allows the student network to learn the softening probability distribution of the teacher network more accurately.

[0034] KL divergence is an asymmetric measure used to measure the difference between two probability distributions. The KL divergence minimization model is used to estimate the difference in predicted loss between the teacher network and the student network.

[0035] ; ; in, For KL divergence loss, Let KL divergence be the KL divergence. The softening probability distribution output by the student network. The softening probability distribution output by the teacher network. This is achieved by introducing a temperature parameter. To scale the loss and prevent the vanishing gradient problem during training, KL divergence loss guides the output of the student network to gradually align with the output distribution of the teacher network.

[0036] To further improve the performance of the student network, this invention also designs a mask feature distillation mechanism. Because the masking of the student network results in a difference in dimension between its feature vector and the teacher network's feature vector, it is necessary to distill the teacher features... Perform bilinear interpolation downsampling, then combine with student features Calculate L2 loss: Because the student network has smaller features than the teacher network due to masking, the teacher features need to be adapted. ; in, For mask feature loss, The staff-like characteristics of the teacher network, For the staff characteristics of student networks, This represents a bilinear interpolation operation used to downsample teacher features to the dimension of student features.

[0037] Finally, by weighting and summing the KL divergence loss and the mask feature loss separately, the total knowledge distillation loss is obtained: ; in, The total loss from knowledge distillation. The weights for the KL divergence loss are... The weights are the mask feature loss values.

[0038] Step 5: Fix the parameters of the Transformer backbone network of the trained student network, and add a ResNet model between the Transformer backbone network and the softmax activation function. The role of ResNet in the network is to solve the performance bottleneck caused by increasing the number of model layers by introducing residual blocks, allowing the network to deepen the number of layers more stably. Then train a lightweight classification model to complete the music staff classification task, such as... Figure 1 As shown in b.

[0039] Figure 2 The results show that, under the same number of training iterations, the method of this invention significantly improves the recognition rate compared to the baseline model. The core of this invention lies in employing an online distillation method, using temperature softening and KL divergence loss to jointly train the model. This innovative loss design stems from the observation of the synergistic effect of global and local information in speech emotion expression. Initializing the teacher-student network through a pre-trained model provides a solid representational foundation for subsequent self-supervised learning, thereby effectively promoting the optimization of the online distillation process.

[0040] A second embodiment of the present invention provides a music staff notation emotion classification system, comprising: The data acquisition module is used to acquire and preprocess musical staff data; The vector conversion module is used to convert musical staves into musical stave data vectors; The feature extraction module uses a long short-term memory network to extract features from the five-line staff. The knowledge distillation module enables knowledge transfer from the teacher network to the student network. The classification module is used to complete the final sentiment classification task.

[0041] Furthermore, the knowledge distillation module includes a teacher network and a student network. The teacher network consists of a long short-term memory network, a multi-level Transformer backbone network, and a softmax activation function. The student network adds a masking module to the teacher network.

[0042] A third embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a music staff emotion classification method based on multi-level distillation.

[0043] A fourth embodiment of the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement a music staff emotion classification method based on multi-level distillation.

[0044] The above description merely illustrates preferred embodiments of the present invention and is not intended to limit the invention in any other way. Any person skilled in the art may make modifications or alterations to the above-disclosed content to create equivalent embodiments. However, any simple modifications, equivalent changes, and alterations made to the above embodiments based on the technical essence of the present invention, without departing from the scope of the present invention, shall still fall within the protection scope of the present invention.

Claims

1. A music staff notation emotion classification method based on multi-level distillation, characterized in that, Includes the following steps: Step 1: Collect musical staff data and construct a training dataset containing various emotional types; Step 2: Convert the staff notation into a staff notation data vector, including embedding and splicing pitch and dynamics. Step 3: Use a long short-term memory network to extract features from the staff data vector to obtain staff features; Step 4: Construct teacher and student networks to perform knowledge distillation, and achieve knowledge transfer through temperature scaling and KL divergence loss; Step 5: Fix the Transformer backbone network parameters of the student network and add a ResNet model for lightweight classification training.

2. The method according to claim 1, characterized in that, Step 2 specifically includes: The pitch embedding module and the velocity embedding module are used to perform embedding transformations on the pitch data and velocity data respectively, to obtain the pitch embedding vector and the velocity embedding vector respectively; Acquire the pitch and dynamics of each track in each sampling period; Pitch data is obtained by embedding pitch into a pitch vector, and velocity data is obtained by embedding velocity into a velocity vector. After concatenating the pitch data and velocity data, the five-line staff data vector is obtained by passing the concatenated vector through a linear fully connected layer.

3. The method according to claim 1, characterized in that, The long short-term memory network in step 3 includes: Forget gates are used to determine the degree to which information from the previous output position has been forgotten. Input gates are used to update the state of memory cells at the current time step. Output gates are used to control the output value at the current moment. Memory units are used to store long-term dependency information.

4. The method according to claim 1, characterized in that, The knowledge distillation in step 4 includes: The outputs of the teacher network and student network are softened using temperature parameters; KL divergence loss is used to measure the difference in output distribution between the teacher network and the student network. We design a mask feature distillation mechanism that adapts the feature dimension through bilinear interpolation.

5. The method according to claim 4, characterized in that, The teacher network consists of a long short-term memory network, a multi-level Transformer backbone network, and a softmax activation function, while the student network consists of a long short-term memory network, a mask module, a multi-level Transformer backbone network, and a softmax activation function.

6. The method according to claim 1, characterized in that, The total loss of knowledge distillation consists of KL divergence loss and mask feature loss.

7. A musical staff notation emotion classification system for implementing the method of any one of claims 1-6, characterized in that, include: The data acquisition module is used to acquire and preprocess musical staff data; The vector conversion module is used to convert musical staves into musical stave data vectors; The feature extraction module uses a long short-term memory network to extract features from the five-line staff. The knowledge distillation module enables knowledge transfer from the teacher network to the student network. The classification module is used to complete the final sentiment classification task.

8. The system according to claim 7, characterized in that, The knowledge distillation module includes a teacher network and a student network. The teacher network consists of a long short-term memory network, a multi-level Transformer backbone network, and a softmax activation function. The student network adds a masking module to the teacher network.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-6.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory, characterized in that, When the processor executes the computer program, it implements the method as described in any one of claims 1-6.

Images