Symbol music generation method with emotional conditions

Abstract

The invention discloses a method for generating symbol music with emotional conditions, and the method comprises the steps: S1, preparing a dataset which comprises a midi dataset and a piano roll dataset, the piano roll dataset is obtained through the preprocessing of the midi dataset, and the midi dataset and the piano roll dataset both comprise symbol music data containing classical music and popular music; s2, training a VAE network which is used for encoding the piano roll data into a hidden space; s3, pre-training a diffusion model; the diffusion model adopts a DiT architecture, and the diffusion model is trained on the piano roll data with the time sequence length of 10.24 s after segmentation; s4, using random control guidance SCG to carry out forward evaluation on an undifferentiated rule function, and working together with the diffusion model in a plug-and-play mode to realize undifferentiated rule guidance on the diffusion model; s5, setting a music emotion evaluation index MAV; and S6, setting a non-differentiatable rule function with an emotion condition to guide the diffusion model to generate a music clip meeting the emotion condition in a back diffusion process.

Description

Technical Field

[0001] The present application relates to the technical field of deep neural networks, and in particular to a method for generating symbolic music with emotional conditions based on deep neural networks. Background Art

[0002] Music is not only an effective tool for regulating emotions and expressing feelings, but also plays an important role in promoting cognitive function and mental health. With the advancement of technology and the popularization of digital music, music has become extremely convenient to obtain, but it has also brought about a rapid increase in demand for music. This demand is not only reflected in quantity, but also in the improvement of the requirements for refinement and personalization of music. In this context, the research on music generation technology is particularly important. Emotional music generation can not only help people regulate their emotions in specific situations, but also provide new possibilities for creation. Through emotional music generation technology,

[0003] The emotional expression of music can be adjusted more accurately to adapt to personal emotional needs or specific situational requirements. The development of this technology can not only enrich the means of music creation, but also play an important role in psychotherapy, education and entertainment. Therefore, the research and implementation of emotional music generation technology is not only at the forefront of the development of music technology, but also has a positive role in promoting the various needs of society.

[0004] There are currently two approaches to emotion-conditioned signed music generation based on deep learning: The first is to convert emotion labels into embeddings and use them as input to the model (Madhok et al. [1] Use LSTM to generate music emotions and incorporate some emotions represented by faces. Hung et al. [2] The EOMPIA music emotion dataset was proposed. The music was manually segmented and the Russell 4Q emotion model was used to annotate the emotions to ensure that each segment can express consistent emotions. The Transformer model and LSTM model were used to generate music. [3] et al. used the Transformer-XL model to generate music with negative, positive, and neutral emotions. Sulun et al. [4] They trained the transformer model using continuous arousal and valence labels, and used a continuous value concatenation model to combine the emotional condition with each musical token during generation, ensuring that the emotional condition was applied evenly throughout the sequence. Grekow et al. [5] Using conditional autoencoders to train a model to generate music. Xu et al. [6]Extract the polyphonic characteristics and tonality of music as musical emotional features, and decouple the learning of the hidden distribution of the two features in the emotional space through a deep latent variable model. The second type is to train an emotional classifier and apply it to the model output to guide the decoding process or the latent space of the variational autoencoder and the generative adversarial network. Ferreira et al. [7] The hidden state of LSTM is used as feature representation to train a logistic regression model for sentiment classification. Bao and Sun [8] Use BERT to train an automatic emotion recognition model on the Edmonds Dance dataset, and then train an end-to-end encoder-decoder model to generate music. Tan and Herremans [9] We model the corresponding quantifiable low-level attributes to learn high-level feature representations, and infer high-level features from low-level representations through Gaussian mixture variational autoencoders (GM-VAEs), opening up the possibility of learning representations of abstract high-level musical qualities under data-scarce conditions. Tseng et al.

[10] By combining generative adversarial networks (GANs) and variational autoencoders (VAEs) to expand musical works, the model trains two classifiers (one for musical emotion and one for musical tonality) for GAN training. In addition to training classifiers to classify musical emotions, Qin et al.

[11] A multimodal music emotion label generation method is proposed to predict music emotion based on lyrics text data combined with audio mel-spectrogram and EEG spectrogram.

[0005] However, existing methods still have some shortcomings. 1) The emotional labels given by data annotators may be affected by objective and subjective factors, resulting in subjective bias in emotional labels. It is difficult for end-to-end methods to learn the relationship between emotions and music sequences. The accuracy of emotion classifiers is low, and other classification methods require other additional labels such as lyrics, which are not suitable for symbolic music generation. 2) The speed and rhythm related to the arousal of music and emotions are easier to be learned by the model, but the form and frequency of harmonic changes related to the valence of emotions are almost never learned by the model, which causes the music generated according to emotional conditions to be very vague in valence. 3) The modeling of these emotional conditions is still a difficult problem, and basically only 4 types of emotional conditions can be generated. 4) The evaluation of emotions is relatively subjective and the accuracy of the emotion classification task of symbolic music is relatively low, making it difficult to make reliable objective evaluations.

[0006] References:

[0007] [1]Madhok R,Goel S,Garg S.Sentimozart:Music generation based onemotions[C].In Proceedings of International Conference on Agents andArtificial Intelligence,2018:501-506.

[0008] [2]Hung H,Ching J,Doh S,Kim N,Nam J,and Yang Y.EMOPIA:A multi-modalpop piano dataset for emotion recognition and emotion-based music generation[C].In Proceedings of International Society for Music Information RetrievalConference,2021:318–325.

[0009] [3]Pangestu M,Suyanto S.Generating music with emotion usingtransformer[C].In Proceedings of International Conference on Computer Scienceand Engineering,2021:1–6.

[0010] [4]Sulun S,Davies M E P,and Viana P.Symbolic music generationconditioned on continuous-valued emotions[J].IEEE Access,2022,10:44617–44626.

[0011] [5]Grekow J,Dimitrova-Grekow T.Monophonic music generation with agiven emotion using conditional variational autoencoder[J].IEEE Access,2021,9:129088–129101.

[0012] [6] Xu Bei, Liu Tong, Fu Xiaokun, et al. A method and system for generating specific emotional music[P]. Jiangsu Province: CN202310234247.9, 2023-08-18.

[0013] [7]Ferreira L,Whitehead J.Learning to generate music with sentiment[C].In Proceedings of International Society for Music Information RetrievalConference, 2019:384–390.

[0014] [8]Bao C,Sun Q.Generating music with emotions[J].IEEE Transactions onMultimedia,2022,24(8):3602-3614

[0015] [9]Tan HH,Herremans D.Music fadernets:Controllable music generationbased on high-level features via lowlevel feature modeling[C].In Proceedings of International Society for Music Information Retrieval Conference, 2020:109–116.

[0016]

[10] Tseng BW, Shen YL, Chi T S. Extending Music Based On Emotion AndTonality Via Generative Adversarial Network. International Conference on Acoustics, Speech and Signal Processing, 2021.

[0017]

[11] Qin Yong, Wang Xuechen, Li Qicheng, et al. A method for generating music emotion labels based on multimodal fusion[P]. Tianjin: CN202211480294.3, 2023-03-07.

[0018]

[12] Kang C, Lu P, Yu B, Tan Summary of the invention

[0019] The purpose of the present invention is to overcome the shortcomings of the prior art and provide a method for generating symbolic music with emotional conditions, which generates music clips with higher accuracy according to multiple music emotional conditions (more than four categories), reduces subjective bias, and improves the accuracy of valence. The method also includes a new objective evaluation method (index) for music emotion.

[0020] The objective of the present invention is achieved through the following technical solutions:

[0021] A method for generating symbolic music with emotional conditions, comprising:

[0022] S1. Prepare a data set, which includes a midi data set and a piano roll data set. The piano roll data set is obtained by preprocessing the midi data set. Both the midi data set and the piano roll data set contain symbolic music data covering classical music and popular music. The piano roll data set is obtained by preprocessing the midi data set. Specifically, the midi data is first cut into 10.24 second segments, and then the pitch, note start and pedal in the midi data are obtained to form the piano roll data.

[0023] S2. Train a VAE network to encode piano roll data (3×128×1024) into latent space (4×16×128); in the piano roll data (3×128×1024), 3 refers to pitch roll, note start roll, and pedal roll; 128 refers to the pitch range; 1024 is the time series length, i.e., 10.24s;

[0024] S3. Pre-trained diffusion model; the diffusion model adopts the DiT architecture and is trained on the piano roll data with a time series length of 1024 (10.24s) after segmentation;

[0025] S4. Use random control to guide SCG to perform forward evaluation of non-differentiable rule functions, and work with the diffusion model in a plug-and-play manner to achieve non-differentiable rule guidance for the diffusion model;

[0026] S5. Set the music emotion evaluation index MAV, use several music attributes that can affect music emotion to construct a multidimensional emotion space, each multidimensional emotion space contains several multidimensional music emotion spaces, each music emotion is regarded as a music emotion space, and calculate which music segment is in which music emotion space to obtain the emotion score of the music segment; and map the emotion score to the two-dimensional VA model to obtain an objective evaluation index;

[0027] S6. Set a non-differentiable rule function with emotional conditions to guide the diffusion model to generate music clips that meet the emotional conditions during the reverse diffusion process.

[0028] Furthermore, the data sets include MAESTRO data set, Pop1k7 data set, and Pop909 data set; the midi data set is preprocessed to obtain the piano roll data set, specifically, the midi data is first cut into 10.24 second segments, and then the pitch, note start and pedal in the midi data are obtained to form the piano roll data.

[0029] Furthermore, in step S3, in order to generate a music clip of arbitrary length, the score function aggregation mechanism of DiffCollage is used to aggregate the score function of each piano roll data.

[0030] Furthermore, in step S4, random control guidance SCG is introduced in the t step of the reverse diffusion to calculate X t-1 The posterior mean, then randomly sample X t-1 The posterior mean is obtained by n samples arrive Estimate the denoising result of each noisy sample arrive Calculate the sample that minimizes the loss of X0, and the corresponding sample is the result of this step of denoising X t-1 .

[0031] Furthermore, in step S5, a point representing a general emotional attribute is determined for each multidimensional plane through a clustering algorithm, and the similarity between the attributes of the music clip and these general emotional attributes is calculated to obtain the music emotion evaluation index MAV. The calculation formula of the music emotion evaluation index MAV is as follows:

[0032]

[0033] where x j is the music attribute vector of a music clip with a certain type of emotional label, S i is the clustering of music emotions, C i is the vector of music emotion space i, n i is the number of music clips, c ikis the kth element of the vector of the music emotion space i, m is the dimension of the vector of the music emotion space, y k is the kth element of the music clip attribute vector, U i is the score in different music emotion spaces, L is the number of music emotion spaces, μ i is the mean, σ i is the standard deviation, X i is the eigenvector of the music emotion space, β i is the music emotion space weight.

[0034] Furthermore, in step S5, jSymbolic is first used to select music attributes that are highly correlated with music emotions (Kang C, Lu P, Yu B, Tan X, Ye W, Zhang S, Bian J. EmoGen: Eliminating Subjective Bias in Emotional Music Generation. arXiv preprint arXiv: 2307.01229, 2023.) to form a multidimensional emotional space; the music attribute vectors of the music clips in the midi dataset are calculated and then clustered to obtain different music emotion spaces; the central value of the music emotion space is obtained by calculating the value of the cluster center and representing the general emotion, and the generation process of symbolic music is separated from the emotion label to avoid the subjective bias of the emotion label and better control the generation of emotional music. The generation process of symbolic music is also the process of reverse diffusion; to calculate the music emotion evaluation index MAV of a music clip, it is necessary to first calculate its music attribute vector X, and then the score of the music attribute vector X in different music emotion spaces, and the score is mapped to the two-dimensional VA model to obtain an objective evaluation index.

[0035] Furthermore, in step S6, the non-differentiable rule function with sentiment conditions includes three parts:

[0036] First, after the backward diffusion reaches step number T, emotional guidance is added to the backward diffusion process of the diffusion model. The prediction results generated by each step of the diffusion model are decoded into midi data through the trained VAE network, where the prediction results generated by each step of the diffusion model are piano roll data; the range of step number T is 100-150 to take into account both training time and emotional guidance;

[0037] Then use jSymbolic to obtain the various music attributes of the midi data and calculate the music attribute vector X;

[0038] Finally, the emotional score loss I of the music attribute vector X is calculated, and the formula is defined as follows

[0039]

[0040] c ik is the kth element of the vector of music emotion space i, x k is the kth element of the music attribute vector X, L is the number of music emotion spaces, m is the dimension of the music emotion space vector, γ i is the guiding weight of the music emotion space i.

[0041] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method for generating symbolic music with emotional conditions when executing the program.

[0042] The present invention also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the method for generating symbolic music with emotional conditions are implemented.

[0043] Compared with the prior art, the technical solution of the present invention has the following beneficial effects:

[0044] 1. Improved generation accuracy: The generation method based on the diffusion model can better capture the detailed characteristics of music under emotional conditions. The generated music is more accurate in terms of valence and arousal. It improves the diffusion model's ability to learn valence features and can generate symbolic music with higher quality and authenticity, significantly reducing emotional ambiguity.

[0045] 2. Diversification of emotion modeling: It supports modeling of more than four emotion categories and can flexibly (adding new conditions does not require retraining the network model) generate symbolic music according to a variety of musical emotion conditions and other musical conditions; it greatly improves the flexibility and adaptability of emotion generation and breaks through the limitations of traditional models for emotion classification.

[0046] 3. Reduction of subjective bias: By introducing the emotional black box rule function and the objective evaluation index MAV based on the multidimensional emotional space, the gap of lack of objective analysis of music emotion is filled, which facilitates music emotion-related tasks; this index is more in line with human emotional cognition, eliminates the subjective bias of music emotion labels, and improves the emotional consistency and credibility of generated music.

[0047] 4. Efficiency of generation control: Using random control guidance (SCG) technology and a plug-and-play approach, the diffusion model only needs to (quickly) guide the emotional conditions during the reverse diffusion process, which greatly reduces the time and computer resources consumed by retraining, thereby improving generation efficiency and effectiveness.

[0048] 5. Scientific nature of emotional evaluation: The present invention introduces an objective evaluation index MAV based on a multidimensional emotional space to conduct emotional evaluation on the generated music, ensuring the objectivity and quantitative characteristics of the evaluation results, and providing a reliable basis for subsequent optimization.

[0049] 6. Wide application scope: The dataset covers classical music and popular music, and can generate music clips of various styles, providing solid support for application needs in different fields (such as psychotherapy, education, entertainment, etc.).

[0050] 7.Flexibility in generating length: The DiffCollage scoring function aggregation mechanism is used to generate music clips of arbitrary length, further meeting the needs of personalized and complex music.

[0051] In summary, the present invention is superior to the existing technologies in terms of accuracy, diversity and objective evaluation of emotional expression, and provides new ideas and practical approaches for the field of emotional music generation.

[0052] It proposes an objective evaluation index for music emotion that can be used in deep neural networks. It also eliminates subjective bias. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 It is a flow chart of the method for generating symbolic music with emotional conditions of the present invention.

[0054] Figure 2 It is a symbolic music generation process based on the diffusion model; the diffusion model includes the forward diffusion process and the reverse diffusion process. Figure 2 The forward diffusion is the dashed arrow pointing to the left, and the reverse diffusion is the solid arrow pointing to the right.

[0055] Figure 3 The specific back-diffusion process is shown using random control guidance and introducing a non-differentiable rule function with sentiment conditions.

[0056] Figure 4 It is the result graph generated by the method of the present invention. DETAILED DESCRIPTION

[0057] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0058] The method for generating symbolic music with emotional conditions provided in this embodiment is shown in Figure 1 ,include:

[0059] S1. Prepare the data set: The data set includes a midi data set and a piano roll data set. The piano roll data set is obtained by preprocessing the midi data set. Both the midi data set and the piano roll data set contain symbolic music data covering classical music and pop music to generate music of various styles. The MAESTRO data set is midi data recorded by contestants in international piano competitions, containing approximately 1,200 classical music pieces totaling approximately 200 hours. The Pop1k7 data set contains 1,700 midi versions of pop music totaling 108 hours. The Pop909 data set contains 909 midi versions of pop music totaling 60 hours. The piano roll data set is obtained by preprocessing the midi data set, specifically by first cutting the midi data into 10.24 second segments, and then obtaining the pitch, note start, and pedal in the midi data to form the piano roll data;

[0060] S2. Train a VAE network to encode piano roll data (3×128×1024) into latent space (4×16×128) to better train the diffusion model;

[0061] S3. Pre-trained diffusion model; the diffusion model uses the DiT architecture. Compared with the standard U-Net, the transformer as the backbone network is better at processing hidden token sequences. This embodiment trains the diffusion model on piano roll data with a time series length of 1024 (10.24s) after segmentation. In order to generate music clips of arbitrary length, the score function aggregation mechanism of DiffCollage is used to aggregate the score function of each piano roll data.

[0062] S4. Use random control to guide SCG to perform forward evaluation of non-differentiable rule functions, and work with the diffusion model in a plug-and-play manner to achieve non-differentiable rule guidance for the diffusion model;

[0063] Since the musical conditions of conditional music generation are usually expressed in the symbolic form of note features, such as note density or chord progression, many of which are non-differentiable, this poses a challenge when using them for guided diffusion. In particular, abstract emotional conditions make it very difficult to calculate the gradient in reverse diffusion. This embodiment uses random control guidance SCG, which is a guidance method that does not require the calculation of gradients. It only requires forward evaluation of the rule function and can work with the diffusion model in a plug-and-play manner, thereby achieving non-differentiable rule guidance of the diffusion model. Among them, the rule function is the concept of SCG. The role of the rule function for random control guidance is equivalent to the role of the loss function for loss guidance. Generally, the loss function must be differentiable. The rule function in this embodiment is generally not differentiable, so it is called a non-differentiable rule function.

[0064] like Figure 2 , Figure 3 As shown, SCG random control guidance is introduced in the t step of back diffusion to calculate X t-1 The posterior mean, then randomly sample X t-1 The posterior mean is obtained by n samples arrive Estimate the denoising result of each noisy sample arrive Calculate the sample that minimizes the loss of X0, and the corresponding sample is the result of this step of denoising X t-1 .

[0065] S5. Set the music emotion evaluation index MAV:

[0066] For music, how to evaluate the emotion of music is an important issue. At present, the evaluation of music emotion is mostly based on people's subjective evaluation. This embodiment designs a music emotion evaluation index MAV (Music Attribute Value), which uses several music attributes that can affect music emotion to construct a multidimensional emotional space. Each multidimensional emotional space contains several multidimensional music emotion spaces. Each music emotion is regarded as a music emotion space. By calculating which music emotion space the music clip is in, the emotional score of the music clip is obtained; but due to people's subjective bias, these emotional multidimensional planes will intersect. At this time, the same music clip will feel different emotions in different people, different moods, etc. In order to eliminate the influence of this bias, a point representing a general emotional attribute is determined in each multidimensional music emotion space through a clustering algorithm, and the similarity (loss) between the attributes of the music clip and these general emotional attributes is calculated to obtain the score of the music emotion evaluation index MAV. The MAV formula is as follows:

[0067]

[0068]

[0069] where x j is the music attribute vector of a music clip with a certain type of emotional label, S i is the clustering of music emotions, C i is the vector of music emotion space i, n i is the number of music clips, c ik is the kth element of the vector of the music emotion space i, m is the dimension of the vector of the music emotion space, y k is the kth element of the music clip attribute vector, U i is the score in different music emotion spaces, L is the number of music emotion spaces, μ i is the mean, σ i is the standard deviation, X iis the eigenvector of the music emotion space, β i is the music emotion space weight.

[0070] Specifically: First use jSymbolic to select the most relevant

[12] The music attributes of the music clips in the MIDI data set form a multidimensional emotional space. The music attribute vectors of the music clips in the MIDI data set are calculated and then clustered to obtain different music emotional spaces. The central value of the music emotional space is obtained by calculating the value of the cluster center. This central value can represent general emotions. The generation process of symbolic music is separated from the emotional label, which can avoid the subjective bias of the emotional label, thereby better controlling the generation of emotional music. The generation process of symbolic music is also the process of reverse diffusion. To calculate the MAV of a music clip, it is necessary to first calculate its music attribute vector, and then calculate the score of the music attribute vector in different music emotional spaces. The score is mapped to the two-dimensional VA model to obtain an objective evaluation index.

[0071] The objective evaluation index MAV can be combined with subjective evaluation in form, intuitively feeling the connection between music emotion score and specific emotion, which is useful for music emotion related work. It solves the problem that there are few music datasets with emotion labels, emotion labeling is difficult and there is subjective bias.

[0072] S6. Setting a non-differentiable rule function with emotional conditions to guide the diffusion model to generate music clips that meet the emotional conditions during the reverse diffusion process. In this embodiment, the non-differentiable rule function with emotional conditions is a black box rule function, which is different from a general non-differentiable rule function and is a rule function for emotional conditions.

[0073] First, after the reverse diffusion reaches a certain number of steps T, the diffusion model reverse diffusion process adds emotional guidance, and the prediction results (piano roll data) generated by each step of the diffusion model are decoded into midi data through the pre-trained VAE network. If the number of steps T is too high, the music clips generated by the pre-trained diffusion model still contain more noise and cannot effectively guide emotions. If T is set too low, the emotional guidance effect is not obvious. After many experiments, this embodiment sets T to 100-150 to take into account both training time and emotional guidance.

[0074] Then use jSymbolic to obtain the various music attributes of the midi data and calculate the music attribute vector X.

[0075] Finally, the emotional score loss I of the music attribute vector X is calculated, and the formula is defined as follows

[0076]

[0077] c ik is the kth element of the vector of music emotion space i, xk is the kth element of the music attribute vector X, L is the number of music emotion spaces, m is the dimension of the music emotion space vector, γ i is the guiding weight of the music emotion space i.

[0078] Specifically, the specific application of the above-mentioned symbolic music generation method with emotional conditions based on deep neural network is as follows:

[0079] First, a VAE model is trained to encode the piano roll data into a latent space, and then a diffusion model with a DiT architecture is trained on the piano roll data. The diffusion model is trained using conditions based on the datasets: MAESTRO and Pop (pop1k7 and pop909), following the no-classifier guidance, with a dropout rate of 0.1. The diffusion model is trained for 1.2M steps and uses DDPM with 1000 steps as the default sampling method unless otherwise stated. All experiments are performed on an NVIDIA V100 GPU.

[0080] Then, in the process of back diffusion, SCG random control guidance is used to solve the problem of non-differentiable emotional conditions. Starting from the 750th step of back diffusion, the gradient is not calculated in each step, but 16 samples are taken first, and then the denoising results of the sampled results are predicted, the regular loss of these results is calculated, and a complete denoising result with the smallest loss is selected, and the sampling of this result is generated as the denoising result of the next step. Starting from the 120th step of back diffusion, the gradient is not calculated in each step, but 16 samples are taken first, and then the denoising results of the sampled results are predicted, and other regular losses of these results are calculated, and then the pre-trained VAE network is used for decoding, and the regular loss is calculated using the non-differentiable regular function with emotional conditions and the denoising result with the smallest total loss is selected, and the sampling of this result is generated as the denoising result of the next step.

[0081] Finally, reverse diffusion obtains symbolic music that meets emotional conditions and other conditions, such as Figure 4 shown.

[0082] This embodiment has been verified through a large number of experiments to verify the effectiveness of the network model and the rule-guided emotional black box, and to verify that the symbolic music generated by the method of the present invention has achieved good results in traditional evaluation methods and in the music emotion evaluation index MAV. It has also been verified that MAV is more in line with human emotional judgment and more convenient than traditional evaluation methods.

[0083] Specific:

[0084] 1) The method of the present invention can flexibly (adding new conditions does not require retraining the network model) generate symbolic music that meets various music emotion conditions and other music conditions; refer to Table 6.

[0085] 2) It can generate symbolic music with high quality and authenticity; refer to Table 1 and Table 2.

[0086] 3) The subjective bias of music emotion labels is eliminated; refer to Table 3, Table 4, and Table 5.

[0087] 4) The learning ability of the network model for valence features is improved; refer to Table 4 and Table 5.

[0088] 5) An objective evaluation index for music emotion that can be used in deep neural networks is proposed. This fills the gap in the lack of objective analysis of music emotion and provides convenience for music emotion-related tasks. This index is more in line with human emotional cognition and also eliminates subjective bias; refer to Table 4, Table 5, and Table 6.

[0089] The method of the present invention is compared with the methods of EMOGEN, EOMPIA and TRANSFORMER-GAN. The objective criteria are pitch range (the distance between the highest point and the lowest point), number of pitch categories and polyphonic pitch, as well as other indicators related to music quality. For the symbolic music generated for each emotional category, this embodiment generates 20*5, a total of 100 samples, of which 20 are emotionless categories, and evaluates these samples based on the above characteristics. The results are then averaged, and the overlap area between the attribute set calculation set and the data set set is obtained as the AOA. The expected value is the AOA calculated by taking a part of the data in the data set as a set and another part of the data as a set. Other methods EMOPIA, emogen, and trans-gan all calculate AOA. The results are shown in Table 1

[0090] Table 1

[0091] EOMPIA EMOGEN TRANSFORMER-GAN Method of the present invention Expected Value AOA 0.903 0.890 0.892 0.919 0.949

[0092] The results of the present invention have higher AOA than other methods EMOPIA, emogen, and trans-gan, which indicates that the data generated by the present invention method is more consistent with the original data.

[0093] There are many subjective evaluations of music quality. The method of the present invention is compared with the methods of EMOGEN, EOMPIA and TRANSFORMER-GAN, and the authenticity index is selected. The authenticity score is 1-5 points. For the symbolic music generated for each emotional category, 20*5, a total of 100 samples are generated, of which 20 are emotionless categories. Other methods have 20 music samples for each method. I also extracted 40 samples from the four data sets of EMOPIALAKH POP909 POP1K7. A total of 200 samples were extracted from which some music samples were asked to be scored for authenticity by 50 people. The scoring results are shown in Table 2

[0094] Table 2

[0095] EOMPIA EMOGEN TRANSFORMER-GAN Method of the present invention Authenticity 3.31 3.51 3.45 3.49 Dataset 4.26 4.26 4.12 4.03 ratio 0.78 0.82 0.84 0.87

[0096] Both eompia and emogen are more inclined to pop music, while trans-gan and the method of the present invention add a large amount of MIDI data of classical music competitions. The subjective authenticity of different data sets is also investigated, and the score of the real data set cannot reach more than 4.5 points. The general score of pop music is higher than that of classical music. If the ratio of the authenticity score of the generated music sample and the data set is calculated, the method of the present invention will be higher than other methods, indicating that the generated music simulates the data set to a higher degree.

[0097] Table 3

[0098] EOMPIA EMOGEN TRANSFORMER-GAN Method of the present invention Accuracy(*) 0.418 — — — Accuracy(**) 0.433 0.650 — — Accuracy 0.448 0.673 0.642 0.703

[0099] In the subjective evaluation of music emotion of the present invention, participants need to give specific valence and arousal scores, which are mapped to different emotional spaces, so as to reduce the influence of subjective bias. The scores of valence and arousal are both 1-5 points. The samples and participants are the same as above. The scoring results are shown in Table 3. The accuracy rate (*) is the highest emotional accuracy rate of the emotional conditional music generation results obtained by the subjective evaluation survey conducted by the author of emopia, and the accuracy rate (**) is the highest emotional accuracy rate of the emotional conditional music generation results obtained by the author of emogen through a subjective evaluation survey. The accuracy rate is the subjective evaluation survey result of the present invention. Due to various reasons such as subjective bias, the results of different survey groups will have some differences. The accuracy rate (*) and accuracy rate (**) are for reference of the results of the present invention. The emotional accuracy of the emotional conditional music generation results of the present invention is slightly higher than that of emogen and trans-gan. The accuracy rate of the survey group's judgment of emotion is also slightly higher than the results of other methods as a whole, which is within a reasonable range.

[0100] Objective evaluation of music emotion usually uses music emotion classifier. Emopia trained a linear transformer emotion classifier and achieved good results as an objective evaluation method for most subsequent methods. The present invention proposes an objective evaluation index for music emotion, MAV. The beneficial effects of MAV are shown in the table.

[0101] Table 4

[0102] EOMPIA MAV 4Q 0.684 0.731 A 0.822 0.900 V 0.833 0.870

[0103] The objective evaluation method of the present invention is more objective than the emopia method, and the accuracy of the potency is significantly improved, which alleviates the potency ambiguity to a certain extent.

[0104] In Table 5, the results of the present invention have achieved higher results on L-trans and MAV compared with other methods EMOPIA, emogen, and trans-gan. Among them, since the black box loss guidance used in the back diffusion process is similar to the MAV calculation, the accuracy on MAV is higher. It is worth noting that other eompia and emogen methods have achieved better results on MAV, because MAV is more in line with the characteristics of the data itself than the L-trans classifier. The verification experiments of the two methods on the data set have confirmed this phenomenon. MAV has more accurate sentiment classification of the data set, and reduces the influence of subjective bias, and is more accurate in judging classic music emotion fragments. Moreover, the MAV method provides a new emotional space classification for the fuzzy parts in the 4Q or VA model, such as data that looks like both Q1 and Q2.

[0105] Table 5

[0106] EOMPIA EMOGEN TRANSFORMER-GAN Method of the present invention 4Q(L-trans) 0.418 0.715 0.703 0.740 A(L-trans) 0.690 0.892 0.880 0.885 V(L-trans) 0.583 0.761 0.779 0.800 4Q(MAV) 0.548 0.723 0.691 0.793 A(MAV) 0.701 0.871 0.859 0.932 V(MAV) 0.623 0.770 0.781 0.891

[0107] Previous work can only generate 4Q emotional conditions. The method of the present invention can generate any emotional conditions. Taking 8Q as an example, the fuzzy emotions between Q1 and Q2, Q2 and Q3, Q3 and Q4, and Q4 and Q1 are divided into Q12, Q23, Q34, and Q41. The accuracy of the generated results is calculated using the MAV indicator, as shown in Table 3-6. 8Q (strict) means that the generated results strictly belong to a certain type of emotion and are correct, and 8Q (loose) means that the generated results fall into the adjacent emotional space and are correct. The experimental results show that 8Q (loose) has a higher accuracy rate and can effectively generate corresponding music based on subtle emotions, but 8Q (strict) has a lower accuracy rate and the accuracy of subtle emotions in music still needs to be improved.

[0108] Table 6

[0109] MAV 8Q(Strict) 0.447 8Q(loose) 0.859

[0110] Preferably, the embodiments of the present application also provide a specific implementation of an electronic device capable of implementing all steps in the method for generating symbolic music with emotional conditions in the above embodiment, and the electronic device specifically includes the following contents:

[0111] Processor, memory, communications interface and bus;

[0112] Among them, the processor, memory, and communication interface communicate with each other through the bus; the communication interface is used to realize information transmission between related devices such as server-side devices, metering devices, and user-side devices.

[0113] The processor is used to call the computer program in the memory, and when the processor executes the computer program, all the steps in the method for generating symbolic music with emotional conditions in the above embodiment are implemented.

[0114] An embodiment of the present application also provides a computer-readable storage medium capable of implementing all the steps of the method for generating symbolic music with emotional conditions in the above-mentioned embodiment. A computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, all the steps of the method for generating symbolic music with emotional conditions in the above-mentioned embodiment are implemented.

[0115] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the hardware + program embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0116] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0117] Although the present application provides method operation steps such as embodiments or flow charts, more or fewer operation steps may be included based on conventional or non-creative labor. The order of steps listed in the embodiments is only one way of executing the order of many steps and does not represent the only execution order. When the actual device or client product is executed, it can be executed in the order of the method shown in the embodiments or the drawings or in parallel (for example, in a parallel processor or multi-threaded processing environment).

[0118] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0119] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0120] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0121] The present invention is not limited to the embodiments described above. The above description of the specific embodiments is intended to describe and illustrate the technical solution of the present invention. The above specific embodiments are merely illustrative and not restrictive. Without departing from the scope of the present invention and the scope of protection of the claims, a person of ordinary skill in the art can also make many forms of specific changes under the guidance of the present invention, which all fall within the scope of protection of the present invention.

Claims

1. A method for generating symbolic music with emotional conditions, characterized in that: include: S1. Prepare a data set, which includes a midi data set and a piano roll data set. The piano roll data set is obtained by preprocessing the midi data set. Both the midi data set and the piano roll data set contain symbolic music data covering classical music and popular music. S2. Train a VAE network to encode piano roll data into latent space; S3. Pre-trained diffusion model; the diffusion model adopts the DiT architecture and is trained on the piano roll data with a time series length of 10.24s after segmentation; S4. Use random control to guide SCG to perform forward evaluation of non-differentiable rule functions, and work with the diffusion model in a plug-and-play manner to achieve non-differentiable rule guidance for the diffusion model; S5. Set the music emotion evaluation index MAV, use several music attributes that can affect music emotion to construct a multidimensional emotion space, each multidimensional emotion space contains several multidimensional music emotion spaces, each music emotion is regarded as a music emotion space, and calculate which music segment is in which music emotion space to obtain the emotion score of the music segment; and map the emotion score to the two-dimensional VA model to obtain an objective evaluation index; S6. Set a non-differentiable rule function with emotional conditions to guide the diffusion model to generate music clips that meet the emotional conditions during the reverse diffusion process.

2. The method for generating symbolic music with emotional conditions according to claim 1, characterized in that: The data sets include MAESTRO data set, Pop1k7 data set, and Pop909 data set. The piano roll data set is obtained by preprocessing the midi data set. Specifically, the midi data set is first cut into 10.24 second segments, and then the pitch, note start and pedal in the midi data are obtained to form the piano roll data.

3. The method for generating symbolic music with emotional conditions according to claim 1, characterized in that: In step S3, in order to generate a music clip of arbitrary length, the score function aggregation mechanism of DiffCollage is used to aggregate the score function of each piano roll data.

4. The method for generating symbolic music with emotional conditions according to claim 1, characterized in that: In step S4, the SCG random control guidance is introduced in the t step of the back diffusion to calculate X t-1 The posterior mean, then randomly sample X t-1 The posterior mean is obtained by n samples arrive Estimate the denoising result of each noisy sample arrive Calculate the sample that minimizes the loss of X0, and the corresponding sample is the result of this step of denoising X t-1 .

5. The method for generating symbolic music with emotional conditions according to claim 1, characterized in that: In step S5, a point representing a general emotional attribute is determined for each multidimensional plane through a clustering algorithm, and the similarity between the attributes of the music clip and these general emotional attributes is calculated to obtain the music emotion evaluation index MAV. The calculation formula of the music emotion evaluation index MAV is as follows: where x j is the music attribute vector of a music clip with a certain type of emotional label, S i is the clustering of music emotions, C i is the vector of music emotion space i, n i is the number of music clips, c ik is the kth element of the vector of the music emotion space i, m is the dimension of the vector of the music emotion space, y k is the kth element of the music clip attribute vector, U i is the score in different music emotion spaces, L is the number of music emotion spaces, μ i is the mean, σ i is the standard deviation, X i is the eigenvector of the music emotion space, β i is the music emotion space weight.

6. A method for generating symbolic music with emotional conditions according to claim 1 or 5, characterized in that: In step S5, jSymbolic is first used to select music attributes that are highly correlated with music emotions to form a multidimensional emotion space; the music attribute vectors of the music clips in the midi data set are calculated and then clustered to obtain different music emotion spaces; the central value of the music emotion space is obtained by calculating the value of the cluster center and represents the general emotion, and the generation process of symbolic music is separated from the emotion label to avoid the subjective bias of the emotion label and better control the generation of emotional music. The generation process of symbolic music is also the process of reverse diffusion; to calculate the music emotion evaluation index MAV of a music clip, it is necessary to first calculate its music attribute vector X, and then the score of the music attribute vector X in different music emotion spaces, and the score is mapped to the two-dimensional VA model to obtain an objective evaluation index.

7. The method for generating symbolic music with emotional conditions according to claim 1, characterized in that: In step S6, the non-differentiable rule function with sentiment conditions includes three parts: First, after the backward diffusion reaches step number T, emotional guidance is added to the backward diffusion process of the diffusion model. The prediction results generated by each step of the diffusion model are decoded into midi data through the trained VAE network, where the prediction results generated by each step of the diffusion model are piano roll data; the range of step number T is 100-150 to take into account both training time and emotional guidance; Then use jSymbolic to obtain the various music attributes of the midi data and calculate the music attribute vector X; Finally, the emotional score loss I of the music attribute vector X is calculated, and the formula is defined as follows c ik is the kth element of the vector of music emotion space i, x k is the kth element of the music attribute vector X, L is the number of music emotion spaces, m is the dimension of the music emotion space vector, γ i is the guiding weight of the music emotion space i.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the method for generating symbolic music with emotional conditions as described in any one of claims 1 to 7 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method for generating symbolic music with emotional conditions as described in any one of claims 1 to 7 are implemented.

Images