Emotional score and music generation method, medium and device based on multi-modal data
By using multimodal data fusion and the CPKT-Transformer model, music with multidimensional emotional features is generated, which solves the problems of monotony and poor listenability when generating music from single-modal data, and achieves richer emotional expression and efficient computation.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SOUTH CHINA UNIV OF TECH
- Filing Date
- 2023-02-28
- Publication Date
- 2026-06-02
AI Technical Summary
Existing deep learning-based music generation methods use single-modal data, resulting in monotonous and poorly listenable music that fails to accurately express individual emotional differences, lacks music theory knowledge, and lacks diversity and authenticity.
Using multimodal data fusion technology, the CPKT-Transformer model is used to combine multiple modal data such as facial expressions, EEG, text, and audio to generate multidimensional emotional features. Taking into account factors such as musical tonality, time signature, chords, and tempo, emotional scores are generated and music is synthesized.
The generated music is richer and more emotionally expressive, with high computational efficiency and support for parallel processing. It solves the problems of redundancy and low computational efficiency when generating music from single-modal data, and improves the listenability and diversity of the music.
Smart Images
Figure CN116312427B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of affective computing technology, and more specifically, to an affective musical score and music generation method, medium, and device based on multimodal data. Background Technology
[0002] Music is a medium for emotional expression, reflecting an individual's feelings. With the rapid development of deep learning, applications of neural networks are flourishing across various industries. Deep learning also holds great potential in the field of music generation; therefore, deep learning models can be built to generate music that expresses emotional characteristics and states.
[0003] Existing deep learning-based music generation methods can be divided into two categories: music generation methods based on non-physiological data and music generation methods based on physiological data.
[0004] Methods for generating music based on non-physiological data, particularly those based on images, facial expressions, and text, generally involve inputting unimodal non-physiological signal data into a deep learning model for training before generating music. These methods have several drawbacks: First, using unimodal non-physiological data results in monotonous music that fails to capture the diversity of music. Second, the emotions evoked by individuals from the same or different emotional stimuli vary greatly, leading to inaccurate model learning and expression of emotions. Third, because physiological data such as facial expressions and body language can be manipulated and disguised, resulting in emotional concealment, the emotions expressed by music generated based on this data are easily confused and misjudged, casting doubt on their authenticity. Fourth, existing methods do not utilize music theory knowledge during music generation, resulting in poor listenability. While a small number of emotion-based music generation methods synthesize music by integrating user-specified emotions into musical elements, the resulting music lacks sufficient mapping to emotions, is rigid, and fails to reflect the differences in emotional music between individuals.
[0005] Music generation methods based on physiological data benefit from the inherent emotional immutability of human physiological data; emotions cannot be artificially disguised. However, while existing methods utilize physiological data and possess favorable emotional characteristics, they all use single-modality physiological signal data as input to deep learning models. A single modality is insufficient to accurately represent an individual's emotional characteristics, thus failing to generate music that accurately reflects those emotions. Furthermore, these methods do not generate emotional musical scores or utilize musical characteristics, resulting in music with poor listenability.
[0006] The two methods mentioned above mostly use ordinary convolutional neural networks (CNNs) or recurrent neural networks (RNNs), which are not optimal solutions. Ordinary CNNs perform poorly in extracting features from shallow layers, while deep CNNs are prone to problems such as large parameter sets, overfitting, and information decay during gradient descent. RNNs are computationally intensive and inefficient, making them unsuitable for music generation algorithms. Furthermore, all these methods use only single-modality data for music generation, which is insufficient to express all emotional features. Fusing emotional features from multiple emotional modalities can complement each other, maximizing emotional features in the music generation module. Summary of the Invention
[0007] To overcome the shortcomings and deficiencies of the existing technology, the purpose of this invention is to provide an emotional musical score and music generation method, storage medium and device based on multimodal data; the method can generate music with more comprehensive emotional characteristics by fusing data from multiple modalities, so that the music contains richer and more multi-dimensional emotional tendency information.
[0008] To achieve the above objectives, the present invention is implemented through the following technical solution: a method for generating emotional musical scores and music based on multimodal data, comprising the following steps:
[0009] Acquire multimodal data; the multimodal data includes two or more unimodal data.
[0010] Sentiment recognition is performed on each single-modal data separately to obtain sentiment prediction results; after multimodal fusion of the sentiment prediction results of all single-modal data, sentiment features are obtained, and then a sentiment instruction with multiple modal and multidimensional sentiment features is generated.
[0011] The emotional command is input into the CPKT-Transformer model; the CPKT-Transformer model includes: a CPKT embedding module, a multi-layer Transformer causal attention layer, and a multi-head prediction module that predicts each event in CKPT; the CPKT embedding module samples the emotional command for musical features, and mixes and encodes the emotional command and the sampled musical features to obtain an event group sequence X containing S event groups, X = {X1, X2, ..., X...} S-1 ,X S}, X i Let X be the i-th event group; input the event group sequence X into a multi-layer Transformer causal attention layer to obtain a representation, and finally perform prediction to generate an emotional musical score;
[0012] Based on the emotional musical score, select instruments to play and synthesize music.
[0013] Preferably, in the CPKT-Transformer model, for the event group sequence X = {X1, X2, ..., X...} S-1 ,X S Each event group X in} i Embedding is performed to obtain the embedding vector E. i,j :
[0014] E i,j =Embedding j (X i,j );
[0015] in, Embedding j For the j-th embedding layer, j = 1, 2... 10;
[0016] Then all embedded vectors E i,j After concatenation, the concatenated vector G is obtained after passing through a fully connected layer. i :
[0017] G i =W in (Concat[E i,1 E i,2 ...,E i,10 ]),
[0018] in, W in These are the weights for the fully connected layer;
[0019] Then the concatenated vector G will be... i With position embedding By adding elements one by one, we obtain the embedding representation of the i-th event group.
[0020]
[0021] in, N H The hidden layer dimension of the Transformer's causal self-attention layer;
[0022] Embedding event groups into representations The input is fed into a multi-layer Transformer causal self-attention layer to obtain the contextual representation H of the i-th event group. i :
[0023]
[0024] Causal_Self_Attn() is used to calculate causal attention;
[0025] Based on the context representation H of the i-th event group i Make a prediction for the (i+1)th event group.
[0026] Preferably, the events in the event group include the event group type, the mood, key and time signature of the current music segment, the current position in the entire music segment (measure / position), the chord type formed by the current note, the current tempo, and the pitch, duration and dynamics of the current note.
[0027] Preferably, the multimodal data includes two or more of the following: facial expression images, photos of body movements, non-facial expression images, electroencephalogram (EEG) signals, text, audio, and video.
[0028] Preferably, the method for obtaining sentiment prediction results is:
[0029] When multimodal data includes facial expression images, a residual neural network based on dilated convolution is used to perform facial expression recognition on a pair of facial expression images to obtain emotion prediction results;
[0030] When multimodal data includes photos of body movements, a residual neural network based on dilated convolution is used to perform posture emotion recognition on the body movement photos to obtain emotion prediction results.
[0031] When multimodal data includes non-expression images, a residual neural network based on dilated convolution is used to identify the warm and cool tones of three pairs of non-expression images, and the sentiment prediction result is obtained based on the emotional tendency of the warm and cool tones.
[0032] When multimodal data includes EEG signals, a dynamic graph convolutional neural network is used to perform EEG emotion recognition on the EEG signals to obtain emotion prediction results;
[0033] When multimodal data includes text, a long short-term memory model is used to classify the word vectors of the text as positive, neutral, or negative sentiment to obtain sentiment prediction results.
[0034] When multimodal data includes audio, a bidirectional long short-term memory model with an attention mechanism is used to analyze the pitch, fundamental frequency, energy, and duration of the audio to determine the sentiment tendency and obtain sentiment prediction results.
[0035] When multimodal data includes video, a 3D convolutional neural network is used to identify the video and obtain sentiment prediction results.
[0036] Preferably, the sentiment prediction results of all single-modal data are fused into a multimodal form using any one of the following methods: voting mechanism, averaging method, or weighting method based on channel noise and signal variance.
[0037] Preferably, after generating the emotional score, the emotional score is displayed on the user interface; after synthesizing the music, the music is played.
[0038] A readable storage medium, characterized in that the readable storage medium stores a computer program, which, when executed by a processor, causes the processor to execute the above-described method for generating emotional musical scores and music based on multimodal data.
[0039] A computer device includes a processor and a memory for storing processor-executable programs, characterized in that, when the processor executes the program stored in the memory, it implements the above-described method for generating emotional musical scores and music based on multimodal data.
[0040] Compared with the prior art, the present invention has the following advantages and beneficial effects:
[0041] 1. This invention utilizes multimodal data (including facial expressions, EEG, text, images, audio, video, etc.) as input and employs deep learning methods to construct models that generate various scores and music with emotional tendencies. Compared to music generation methods that use single-modal data, this invention establishes a mapping relationship between data and emotions from multiple modalities, removes redundancy in emotional features between different modalities, and enhances the complementarity of features between modalities. This allows the generation of music with more comprehensive emotional features, making the music contain richer and more multi-dimensional emotional tendency information.
[0042] 2. This invention utilizes the CPKT-Transformer model for music score generation. Compared to existing music generation methods based on recurrent neural networks or ordinary convolutional neural networks, this invention has high computational efficiency, overcomes the problem that the input of the current time step depends on the output of the previous time step and is difficult to parallelize, and supports parallel computing. Furthermore, it uses the Attention mechanism to solve the long-range dependency problem.
[0043] 3. In addition to considering emotional inclination, this invention also takes into account factors that can affect the quality of music, such as tonality, time signature, chords, tempo, and dynamics, making the generated music more emotionally rich and more listenable. Attached Figure Description
[0044] Figure 1 This is a flowchart of the emotional musical score and music generation method based on multimodal data of the present invention;
[0045] Figure 2 This is a schematic diagram of the emotional musical score and music generation method based on multimodal data of the present invention, which uses different deep learning and machine learning methods to construct a model for each type of single-modal data;
[0046] Figure 3 This is a block diagram of the CPKT-Transformer model in the emotional score and music generation method based on multimodal data of this invention. Detailed Implementation
[0047] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.
[0048] Example 1
[0049] A method for generating emotional musical scores and music based on multimodal data, such as Figure 1 As shown, it includes the following steps:
[0050] S1. Acquire multimodal data; the multimodal data includes two or more types of unimodal data; the multimodal data includes two or more types of facial expression images, body movement photos, non-facial expression images, EEG signals, text (a sentence, a paragraph, numbers, etc., one or more emotional nouns), audio, and video.
[0051] S2. Perform sentiment recognition on each single-modal data to obtain sentiment prediction results;
[0052] like Figure 2 As shown, when multimodal data includes facial expression images, a residual neural network based on dilated convolution is used to perform facial expression recognition on a pair of facial expression images to obtain emotion prediction results;
[0053] When multimodal data includes photos of body movements, a residual neural network based on dilated convolution is used to perform posture emotion recognition on the body movement photos to obtain emotion prediction results.
[0054] When multimodal data includes non-expression images, a residual neural network based on dilated convolution is used to identify the warm and cool tones of three pairs of non-expression images, and the emotion prediction result is obtained based on the emotional tendency of the warm and cool tones.
[0055] For facial expression images, photos of body movements, and non-facial expression images, different dilated convolutional neural networks are trained for recognition. Facial expression images and photos of body movements can contain various emotions such as happiness, sadness, and neutrality; non-facial expression images can be images of any content, such as landscapes, and are classified into warm and cool tones using a deep learning model. All three types of data employ residual networks based on dilated convolutions. By expanding the receptive field and adding residual blocks, the model's ability to extract emotional features is enhanced. Convolutional kernels are used to capture emotion-related features in the image, and then a classifier is used to obtain the emotional tendency of the data, thus yielding the emotion prediction result.
[0056] When multimodal data includes EEG signals, a dynamic graph convolutional neural network is used to perform EEG emotion recognition on the EEG signals to obtain emotion prediction results.
[0057] When multimodal data includes text, a long short-term memory model is used to extract sentiment features from the text data. Then, a classifier is used to obtain the sentiment tendency of the text data, and the sentiment prediction result is obtained.
[0058] When multimodal data includes audio, a bidirectional long short-term memory model using the attention mechanism is employed to analyze the pitch, fundamental frequency, energy, and duration of the audio to determine the emotional tendency and obtain the emotional prediction result. Specifically, the attention mechanism is first used to enhance the contextual semantic information and obtain deeper features, and finally a classifier is used to perform emotion recognition to complete the prediction of the corresponding speech emotion.
[0059] When multimodal data includes video, a 3D convolution-based neural network is used to identify the video. Unlike images, videos have four dimensions, so the convolution kernel needs to be 3D convolution. After extracting features through the 3D convolution model, a classifier is connected to determine the sentiment tendency and obtain the sentiment prediction result.
[0060] The emotion recognition model for each modality can be modified or other deep learning or machine learning methods can be adopted according to different use cases.
[0061] Subsequently, the sentiment prediction results of all single-modal data are fused using a multimodal method to obtain sentiment features, thereby generating a sentiment instruction with multiple modalities and dimensions of sentiment features. Specifically, multimodal fusion is performed using any one of the following methods: a voting mechanism, an averaging method, or a weighting method based on channel noise and signal variance.
[0062] S3. Input the emotional command into the CPKT-Transformer model; such as Figure 3 As shown, the CPKT-Transformer model includes: a CPKT embedding module, a multi-layer Transformer causal attention layer, and a multi-head prediction module that predicts each event in CKPT.
[0063] The CPKT embedding module samples musical features from emotional commands, and then mixes and encodes the emotional commands and the sampled musical features (tonality, time signature, chords, tempo, dynamics, etc.) to obtain an event group sequence X containing S event groups, X = {X1, X2, ..., X...}. S-1 ,X S}, X i Let i be the i-th event group. The events in an event group include the event group type, the mood, key, and time signature of the current musical segment, the current position within the entire musical segment, the chord type formed by the current note, the current tempo, and the pitch, duration, and dynamics of the current note. Event group types include mood event groups, rhythm event groups, note event groups, and fill event groups.
[0064] The event group sequence X is input into a multi-layer Transformer causal attention layer to obtain a representation, and finally prediction is performed to generate an emotional score.
[0065] Music places great emphasis on structure, and musical works often exhibit significant repetition in structure and meaning, with this self-correlation manifesting across multiple time scales. Therefore, a Transformer-based structure is chosen to extract contextual representations, which are then processed through fully connected layers to generate musical scores. Multiple fully connected layers are used to generate various types of scores, including but not limited to four-line staff, five-line staff, six-line staff, numbered musical notation, guzheng scores, violin scores, interludes, BD scores, grid scores, shakuhachi scores, curve scores, etc. The generated scores can be used for musical performance.
[0066] Specifically, in the CPKT-Transformer model, for the event group sequence X = {X1, X2, ..., X...} S-1 ,X S Each event group X in} i Embedding is performed to obtain the embedding vector E. i,j :
[0067] E i,j =Embedding j (X i,j );
[0068] in, Embedding j For the j-th embedding layer, j = 1, 2... 10;
[0069] Then all embedded vectors E i,j After concatenation, the concatenated vector G is obtained after passing through a fully connected layer. i :
[0070] G i =W in (Concat[E i,1 E i,2 ...,E i,10 ]),
[0071] in, W in These are the weights for the fully connected layer;
[0072] Then the concatenated vector G will be... i With position embedding By adding elements one by one, we obtain the embedding representation of the i-th event group.
[0073]
[0074] in, N H The hidden layer dimension of the Transformer's causal self-attention layer;
[0075] Embedding event groups into representations The input is fed into a multi-layer Transformer causal self-attention layer to obtain the contextual representation H of the i-th event group. i :
[0076]
[0077] Among them, Causal_Self_Attn() is used to calculate causal attention;
[0078] Based on the context representation H of the i-th event group i Make a prediction for the (i+1)th event group.
[0079] S4. Based on the emotional score, select instruments to play and synthesize music.
[0080] After generating the emotional musical score, it is displayed on the user interface; after synthesizing the music, it is played. Existing music generation methods typically generate music for only one instrument and do not provide sheet music, failing to meet users' needs for self-play. Current sheet music generation methods typically generate sheet music for only one specific instrument, which is rather limited and cannot meet the needs of multi-instrument play. This invention can generate music for multiple instruments, offering greater diversity, and can also generate sheet music for multiple instruments, making it convenient for users to play themselves.
[0081] Example 2
[0082] This embodiment provides a readable storage medium storing a computer program that, when executed by a processor, causes the processor to perform the emotional music score and music generation method based on multimodal data as described in Embodiment 1.
[0083] Example 3
[0084] This embodiment discloses a computer device, including a processor and a memory for storing processor-executable programs. When the processor executes the program stored in the memory, it implements the emotional music score and music generation method based on multimodal data as described in Embodiment 1.
[0085] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.
Claims
1. A method for generating emotional musical scores and music based on multimodal data, characterized in that: Includes the following steps: Acquire multimodal data; the multimodal data includes two or more unimodal data. Sentiment recognition is performed on each single-modal data separately to obtain sentiment prediction results; after multimodal fusion of the sentiment prediction results of all single-modal data, sentiment features are obtained, and then a sentiment instruction with multiple modal and multidimensional sentiment features is generated. The emotional command is input into the CPKT-Transformer model; the CPKT-Transformer model includes: a CPKT embedding module, a multi-layer Transformer causal attention layer, and a multi-head prediction module that predicts each event in the CPKT embedding module; the CPKT embedding module samples the emotional command for musical features, and mixes and encodes the emotional command and the sampled musical features to obtain an event group sequence X containing S event groups, X={ X1, X2,..., X... S-1 X S }, X i Let X be the i-th event group; input the event group sequence X into a multi-layer Transformer causal attention layer to obtain a representation, and finally perform prediction to generate an emotional musical score; Based on the emotional musical score, select instruments to play and synthesize music; In the CPKT-Transformer model, for the event group sequence X={ X1, X2,..., X... S-1 X S Each event group X in} i Embedding is performed to obtain the embedding vector E. i,j : ； in, , For the j-th embedding layer, j=1,2...10; Then all embedded vectors E i,j After concatenation, the concatenated vector G is obtained after passing through a fully connected layer. i : ； in, W in These are the weights for the fully connected layer. Then the concatenated vectors will be... With position embedding By adding elements one by one, we obtain the embedding representation of the i-th event group. : ； in, , For the hidden layer dimensions of the Transformer's causal self-attention layer; Embedding event groups into representations The input is fed into a multi-layer Transformer causal self-attention layer to obtain the contextual representation of the i-th event group. : ,i=1,2...S; Causal_Self_Attn() is used to calculate causal attention; Based on the context representation of the i-th event group Make a prediction for the (i+1)th event group.
2. The emotional musical score and music generation method based on multimodal data according to claim 1, characterized in that: The events in the event group include the event group type, the mood, key and time signature of the current music segment, the current position in the entire music segment, the chord type formed by the current note, the current tempo, and the pitch, duration and dynamics of the current note.
3. The emotional musical score and music generation method based on multimodal data according to claim 1, characterized in that: The multimodal data includes two or more of the following: facial expression images, photos of body movements, non-facial expression images, electroencephalogram (EEG) signals, text, audio, and video.
4. The emotional musical score and music generation method based on multimodal data according to claim 3, characterized in that: The method for obtaining sentiment prediction results is: When multimodal data includes facial expression images, a residual neural network based on dilated convolution is used to perform facial expression recognition on a pair of facial expression images to obtain emotion prediction results; When multimodal data includes photos of body movements, a residual neural network based on dilated convolution is used to perform posture emotion recognition on the body movement photos to obtain emotion prediction results. When multimodal data includes non-expression images, a residual neural network based on dilated convolution is used to identify the warm and cool tones of three pairs of non-expression images, and the sentiment prediction result is obtained based on the emotional tendency of the warm and cool tones. When multimodal data includes EEG signals, a dynamic graph convolutional neural network is used to perform EEG emotion recognition on the EEG signals to obtain emotion prediction results; When multimodal data includes text, a long short-term memory model is used to classify the word vectors of the text as positive, neutral, or negative sentiment to obtain sentiment prediction results. When multimodal data includes audio, a bidirectional long short-term memory model with an attention mechanism is used to analyze the pitch, fundamental frequency, energy, and duration of the audio to determine the sentiment tendency and obtain sentiment prediction results. When multimodal data includes video, a 3D convolutional neural network is used to identify the video and obtain sentiment prediction results.
5. The emotional score and music generation method based on multimodal data according to claim 1, characterized in that: The sentiment prediction results of all single-modal data are fused into a multimodal model using any one of the following methods: voting mechanism, averaging method, or weighting method based on channel noise and signal variance.
6. The emotional musical score and music generation method based on multimodal data according to claim 1, characterized in that: After generating the emotional score, display the score on the user interface; after synthesizing the music, play the music.
7. A readable storage medium, characterized in that, The readable storage medium stores a computer program that, when executed by a processor, causes the processor to perform the emotional score and music generation method based on multimodal data as described in any one of claims 1-6.
8. A computer device comprising a processor and a memory for storing a processor-executable program, characterized in that, When the processor executes the program stored in the memory, it implements the emotional score and music generation method based on multimodal data as described in any one of claims 1-6.