An automated music description generation method
By constructing the MuTeNet model and using convolutional neural networks and Transformer/LSTM to generate music descriptions, the problem of insufficient readability and flexibility of music descriptions in existing technologies is solved, and efficient and automated music description generation is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- NANJING UNIV
- Filing Date
- 2023-02-15
- Publication Date
- 2026-06-09
AI Technical Summary
Existing technologies struggle to generate high-quality, readable, and flexible music descriptions. Tagging methods cannot fully summarize music style features, and sentence template methods rely on manual annotation and lack flexibility.
MuTeNet, a music description generation model, is constructed. It uses pre-trained music and text codecs, convolutional neural networks and Transformer/LSTM for feature extraction and text generation, and combines mean squared error and cross-entropy loss functions to optimize the model, thereby achieving automated music-to-text description.
Generate high-quality, readable, and flexible music descriptions, reduce manual costs, improve the efficiency of music content understanding, and facilitate music retrieval.
Smart Images
Figure CN116259289B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to a description generation method, and more particularly to an automated music description generation method. Background Technology
[0002] Music is an important art form with a powerful emotional appeal and expressive effect, making textual descriptions of music of significant value. A descriptive text of a piece of music allows readers to quickly understand its style, mood, rhythm, and other elements without listening to the entire piece. A large amount of music description text facilitates the quick and easy retrieval of desired music, which is of great value for video soundtracks and music searches. In the internet age, how to provide textual descriptions of massive amounts of music is an important research question.
[0003] Existing technical solutions almost all rely on generating labels for music through music classification models, and then generating descriptions of the music based on these labels. However, the number and scope of labels depend on predefined human input, thus limiting the description results. If few categories are defined, the amount of information that can be described is limited; if many categories are defined, the description becomes overly complex. For example, defining only two categories for rhythm—soothing and energetic—would not adequately describe music that is soothing in the first half and energetic in the second. Conversely, dividing a piece of music into two parts and providing separate descriptions for each part with different categories would be redundant and chaotic for music with a consistent rhythm throughout. Furthermore, besides rhythm, music also possesses stylistic characteristics in terms of mode, mood, genre, and other dimensions; setting separate categories for each dimension would be extremely tedious. The main drawback of label-based methods is their inability to comprehensively summarize various musical styles and selectively provide a piece of music with appropriate detail.
[0004] The paper "Music autotagging as captioning[J].2020" attempts to solve this problem with a multi-label generation method. It uses a convolutional neural network (CNN) to extract features from the music, obtaining a two-dimensional feature vector representation. Then, a Long Short-Term Memory (LSTM) model automatically outputs music descriptive labels based on the music features. The LSTM model outputs labels in order of importance, stopping output when it deems no more labels needed. This results in more flexible descriptions that highlight key musical features. However, the paper only generates labels such as "electronic music" and "guitar," without producing fluent natural language text, making it difficult for humans to read and understand.
[0005] The paper "A Representation Learning Framework for Bi-directional Music-Sentence Retrieval and Generation[J].2020" uses sentence templates to generate fluent sentences by fitting the labels obtained by the model into the templates. However, this method relies on a large number of manually annotated sentence templates, and the number of empty spaces in each sentence template is fixed, which can only accommodate predetermined label types and cannot be compatible with the flexibility of multi-label methods. Therefore, label-based methods always struggle to balance readability and flexibility. Summary of the Invention
[0006] Purpose of the invention: The technical problem to be solved by the present invention is to provide an automated music description generation method to address the shortcomings of the existing technology.
[0007] To address the aforementioned technical problems, this invention discloses an automated music description generation method, comprising the following steps:
[0008] Step 1: Collect music and text data;
[0009] Step 2, construct the music description generation model MuTeNet; the music description generation model includes music and text codecs; the music and text codecs include: music codecs, namely music encoders and music decoders, and text codecs, namely text encoders and text decoders;
[0010] Step 3: Pre-train the music and text codecs in the music description generation model;
[0011] Step 4: Prepare training data and input it into the pre-trained music description generation model for training; the training data is music and its paired text description.
[0012] Step 5: Input the music into the trained model to generate a text description of the music, thus completing the automated music description generation.
[0013] Beneficial effects:
[0014] From a technical perspective, the technical solution of this invention realizes the process of directly generating text descriptions from music without relying on tag generation, thus avoiding the inherent defects of tag-based methods. Compared with pure tag-based music descriptions, this invention is more readable, and compared with tag descriptions based on sentence templates, it is more flexible in sentence structure, and can better generate descriptive text that is human-readable and summarizes the characteristics of music.
[0015] From an application perspective, the technical solution of this invention (1) can automatically generate music descriptions for users, helping them to quickly read and understand the style and content of a piece of music; (2) the trained model can automatically generate descriptions without manual writing, reducing labor costs; (3) it can automatically convert music content into descriptive text, and with the text generated by this technology as an aid, it is easier to retrieve the music you need. Attached Figure Description
[0016] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments, and the advantages of the present invention in the above and / or other aspects will become clearer.
[0017] Figure 1 A schematic diagram of the overall process of this invention.
[0018] Figure 2 A schematic diagram illustrating the process of pre-training a music and text encoder.
[0019] Figure 3 A schematic diagram of the training process for a music description generation model.
[0020] Figure 4 A schematic diagram of the structure of a pre-trained music codec.
[0021] Figure 5 A diagram illustrating the training process of a music description model that uses Transformer for text encoding and decoding and leverages contrastive learning to enhance performance. Detailed Implementation
[0022] This invention proposes an automated method for generating music descriptions. A music and text codec is pre-trained on a large dataset, and then the pre-trained model is trained on a music description dataset. This results in a model that can automatically generate text descriptions for a piece of music, helping users quickly understand its style and content. Furthermore, to address the challenge of learning music encoding, this approach employs two loss function calculation schemes during the training phase to enhance the audio encoder's learning ability, thus contributing to the generation of higher-quality text descriptions.
[0023] An automated music description generation method includes the following steps:
[0024] Step 1: Collect music and text data;
[0025] The music and text data mentioned do not require labeling or pairing; the music can be of any type, and the text can be of any type.
[0026] Step 2, construct the music description generation model MuTeNet (Music-to-Text Network); the music description generation model includes music and text codecs; the music and text codecs include: music codecs, namely music encoders and music decoders, and text codecs, namely text encoders and text decoders;
[0027] Step 3 involves pre-training the music and text codecs in the music description generation model. Specific methods include:
[0028] Step 3-1: The music and text data collected in Step 1 are used as input data and fed into the corresponding encoder to obtain the encoded vector. Specific methods include:
[0029] For music as input data, the Fourier transform is first used to obtain the input music spectrum, and then the input music spectrum is input into the music encoder built by the convolutional neural network (CNN) to obtain a pre-trained music encoding vector with dimension d×T, where d is the output dimension of the convolutional neural network (CNN) and T is the dimension that increases with the length of the input music.
[0030] For text data as input data, it is processed into token encoding after word segmentation and input into the text encoder. The text encoder adopts a neural network model to obtain a pre-trained text encoding vector. The output dimension of the text encoder is the same as the output dimension d of the convolutional neural network CNN.
[0031] Step 3-2: Input the encoded vector into the corresponding decoder for decoding to obtain the restored result. Specific methods include:
[0032] The pre-trained music encoding vector is input into the music decoder built by the convolutional neural network (CNN) to restore the music spectrum; the pre-trained text encoding vector is input into the text decoder to restore the text.
[0033] Step 3-3: Compare the restoration result with the input data described in Step 3-1 and calculate the loss. Specific methods include:
[0034] For the music reconstruction result, i.e., the reconstructed music spectrum, sharpening is performed on both the reconstructed and input music spectra, and then the mean square error (MSE) loss is calculated. Specific methods include:
[0035] Let M[f, T] be an audio spectrum, where f represents the frequency bandwidth of the spectrum (usually a fixed value of 256 or 512), and T represents the time range of the spectrum, which varies with the length of the audio; M[a:-b] represents the submatrix extracted from the spectrum matrix from row a to row b from the end. Then the sharpening function Sharpen(M) is:
[0036] Sharpen(M)=5×M[2:-2, 2:-2]-M[1:-3, 2:-2]-M[3:-1, 2:-2]-M[2:-2, 1:-3]-M[2:-2, 3:-1];
[0037] The loss function is:
[0038]
[0039] in, This represents the audio sharpening result obtained by the music codec.
[0040] For the text reconstruction result, i.e., the reconstructed text, the cross-entropy between the decoded probability distribution in the reconstructed text and the input text data is calculated. Specific methods include:
[0041] Suppose that the word at position i in the input text data is t. i The text codec predicts that the probability of the position being word j is p. i，j Then the cross-entropy loss is:
[0042]
[0043] Where exp(x)=e x This represents a power function with the natural constant e as its base.
[0044] Steps 3-4 involve calculating the loss gradient, performing backpropagation, updating the parameters of the music description generation model, and completing pre-training. Specific methods include:
[0045] The loss calculated in step 3-3 is backpropagated on the model parameters of the music description generation model MuTeNet. Specific methods include:
[0046] Assuming θ is any parameter in the music description generation model MuTeNet, the gradient of the loss with respect to the parameter θ is calculated as follows: Update the model parameters based on the gradient results and the set learning rate α:
[0047]
[0048] Repeat this step to iteratively optimize the model until pre-training is complete.
[0049] Step 4: Prepare training data and input it into the pre-trained music description generation model for training; the training data is music and its paired text description.
[0050] The specific methods for preparing training data include:
[0051] Collect music and corresponding comment texts, divide the music and comment texts into training and validation sets according to a set ratio, and input the training set into the music description generation model for training; input the data in the training set into the model for training in ascending order of text length, specifically including:
[0052] Step 4-1: Input the music from the training set into the music encoder to obtain the music encoding vector;
[0053] Step 4-2: Input the music encoding vector into the text decoder. The text decoding process is the same as in Step 3-2. The text decoder outputs the word probability distribution at each position. Calculate the cross-entropy loss l1 between the generated word probabilities and the corresponding comment text. The text codec converts the description text into text encoding. The obtained text encoding results guide the music encoding training for comparative learning.
[0054] The specific method for training is as follows:
[0055] First, calculate the similarity between different text encodings. <t i , t j >, where t i t is the encoding result of the i-th sentence in the dataset by the text encoder. j This is the encoding result of the j-th sentence in the dataset; then, the contrastive learning loss l2 is calculated between the similarities between music encodings and between text encodings.
[0056] l2=||softmax{ <t i , t j >}-softmax{ <m i m j >}|| 2
[0057] Where, m i m represents the vector encoding obtained by the music encoder for the i-th music in the dataset. j This represents the vector encoding of the j-th music in the dataset, where m represents the same index. i With t i Corresponding to, i.e., t i It is for music m i The text description,
[0058] Calculate the total loss:
[0059] Loss = l1 + l2
[0060] Step 4-3: For the calculated total loss Loss, backpropagation is performed using the same method as in Step 3-4, and the parameters of the music description generation model are updated to complete the training.
[0061] Step 4-4: Use the validation set to evaluate the performance of the music description generation model. That is, input the music in the validation set into the music description generation model and compare the similarity between the output description and the comment text corresponding to the music in the validation set.
[0062] Steps 4-5 involve repeated validation, with the model that yields the best validation result being selected as the final music description generation model.
[0063] Step 5: Input the music into the trained model to generate a text description of the music, thus completing the automated music description generation.
[0064] The text encoder described in step 3-1 uses a neural network model that adapts to sequence modeling, namely Transformer or LSTM.
[0065] The text decoder described in step 3-2 uses a neural network model that adapts to sequence modeling, namely Transformer or LSTM.
[0066] Example:
[0067] Training model (model structure as follows) Figure 5 The left half shows the overall process of generating text descriptions for music (composed of a music encoder and a text decoder), as follows: Figure 1 As shown.
[0068] Step 101: Collect a large amount of music and text data to pre-train the music and text encoder / decoder. The music and text used for pre-training do not require annotation or pairing. The music can be of any genre, and the text can be any human text such as news, encyclopedia entries, novels, or dialogues. A large amount of data can be obtained from the internet. The process of pre-training the music and text encoder is as follows: Figure 2 As shown:
[0069] Step 201: Input the music or text into the corresponding encoder to obtain the encoded vector. For music input, the music spectrum needs to be obtained first through Fourier transform, and then input into the music encoder built by a convolutional neural network (CNN) to obtain a vector of dimension d×T (reference: Music autotagging as captioning[J].2020.), where d is the output dimension of the CNN, and T is the dimension that increases with the length of the input music, such as... Figure 4As shown. The text needs to be segmented into tokens (see: Neural Machine Translation of Rare Words with Subword Units[J].2016), and then input into the text encoder. The text encoder can be a neural network model suitable for sequence modeling, such as Transiormer or LSTM, as long as its output dimension is the same as the CNN output dimension d.
[0070] Step 202: Input the encoded vector into the corresponding decoder to obtain the decoded result. Input the music encoded vector into the CNN music decoder to restore the music spectrum. Input the text encoded result into the text decoder, which is also a neural network model suitable for sequence modeling, such as Transformer or LSTM, to finally restore a piece of text.
[0071] Step 203: Compare the restored result with the actual input and calculate the loss. For the music restoration result, since the main feature of music is melody, and the melodic details are not clear enough after converting to a spectrogram, such as... Figure 4 The diagram shows that sharpening is needed for both the restored result and the actual input spectrum, followed by calculating the mean squared error (MSE) loss. This yields a better representation of the audio features. Specifically, let M[f, T] be an audio segment, and M[a:-b] refer to the submatrix extracted from the spectrum matrix, from row a to row b from the end. Then, the sharpening function Sharpen(M) = 5 × M[2:-2, 2:-2] - M[1:-3, 2:-2] - M[3:-1, 2:-2] - M[2:-2, 1:-3] - M[2:-2, 3:-1], while the loss function... For the text reconstruction result, the cross-entropy between its decoding probability distribution and the real text is calculated. Specifically, let the word at position i in the input sentence be t. i The model predicts that the probability of this position being word j is p. i，j Then the cross-entropy loss
[0072] Step 204: Calculate the loss gradient, perform backpropagation, and update the model parameters. Using the loss calculated in step 203, perform backpropagation on the neural network model parameters θ to calculate the gradient with respect to each parameter. Update model parameters based on gradient results and the set learning rate α. The pre-training process is completed after several rounds of iterative optimization of the model.
[0073] Step 102: Prepare training data for music and its corresponding text descriptions, and input it into the pre-trained codec model for training. First, collect music and its corresponding comment texts from platforms such as music review websites. Divide all data proportionally into training and validation sets, and input the training set into the model for training. To achieve better training results, the training data can be input into the model in ascending order of text length, making the learning difficulty increase from easy to difficult, ultimately resulting in better description performance. The specific training process for the music description generation model is as follows: Figure 3 As shown:
[0074] Step 301: Input the music into the music encoder to obtain the music encoding vector. The music encoder here is the pre-trained music encoder from step 101.
[0075] Step 302: Input the music encoding vector into the text decoder. The text decoding process is the same as in step 203. The text decoder will provide the word probability distribution at each position. Calculate the cross-entropy loss by comparing the generated text probabilities with the true description. Because obtaining high-quality music encoding is more difficult than obtaining text encoding, text encoding results can be used to guide music encoding training through comparative learning. Specifically, for example... Figure 5 As shown by the dashed line, the similarity between different text encodings is first calculated. <t i , t j Then, the contrastive learning loss l2 = ||softmax{ is calculated based on the similarity between music encodings and the similarity between texts. <t i , t j >}-softmax{ <m i m j >}|| 2 The total loss is L1 + L2. By incorporating contrastive learning, the model can learn higher-quality music encoding, thus achieving better detail in the music description. However, since calculating L2 requires the model to perform an additional text encoding, an encoding buffer is needed to remember the encoding results of each music and text interaction, resulting in higher computational resource overhead.
[0076] Step 303: Calculate the loss gradient, perform backpropagation, and update the model parameters. The loss calculated in step 302 is backpropagated over the neural network model parameters θ to calculate its gradient with respect to each parameter. Update model parameters based on gradient results and the set learning rate α.
[0077] Step 304: Evaluate model performance on the validation set. This involves inputting music from the validation set into the model and comparing the similarity between the model's output description and the real text descriptions from the validation set.
[0078] Step 305: Repeat the validation process in multiple rounds of training, and take the model with the best results on the validation set as the final model.
[0079] Step 103: Input the music into the trained model to generate a text description of the music.
[0080] In its specific implementation, this application provides a computer storage medium and a corresponding data processing unit. The computer storage medium is capable of storing a computer program, which, when executed by the data processing unit, can run the invention's description of an automated music description generation method and some or all of the steps in various embodiments. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.
[0081] Those skilled in the art will clearly understand that the technical solutions in the embodiments of the present invention can be implemented using computer programs and their corresponding general-purpose hardware platforms. Based on this understanding, the technical solutions in the embodiments of the present invention, or the parts that contribute to the prior art, can be embodied in the form of computer programs, i.e., software products. These computer program software products can be stored in a storage medium and include several instructions to cause a device containing a data processing unit (which may be a personal computer, server, microcontroller, MUU, or network device, etc.) to execute the methods described in various embodiments or certain parts of the embodiments of the present invention.
[0082] This invention provides an idea and method for automated music description generation. Many methods and approaches exist for implementing this technical solution; the above description is merely a preferred embodiment of the invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of this invention, and these improvements and modifications should also be considered within the scope of protection of this invention. All components not explicitly stated in this embodiment can be implemented using existing technologies.
Claims
1. An automated music description generation method, characterized in that, Includes the following steps: Step 1: Collect music and text data; Step 2, construct the music description generation model MuTeNet; the music description generation model includes music and text codecs; the music and text codecs include: music codecs, namely music encoders and music decoders, and text codecs, namely text encoders and text decoders; Step 3: Pre-train the music and text codecs in the music description generation model; Step 4: Prepare training data and input it into the pre-trained music description generation model for training; the training data is music and its paired text description. Step 5: Input the music into the trained model to generate a text description of the music, thus completing the automated music description generation. In step 1, the music and text data do not require labeling or pairing; the music can be of any type, and the text can be of any type.
2. The automated music description generation method according to claim 1, characterized in that, The music and text encoder in the pre-trained music description generation model described in step 3 includes the following specific methods: Step 3-1: Use the music and text data collected in Step 1 as input data and input it into the corresponding encoder to obtain the encoding vector; Step 3-2: Input the encoded vector into the corresponding decoder for decoding to obtain the restored result; Step 3-3: Compare the restoration result with the input data described in Step 3-1, and calculate the loss; Steps 3-4: Calculate the loss gradient, perform backpropagation, update the parameters of the music description generation model, and complete the pre-training.
3. The automated music description generation method according to claim 2, characterized in that, The specific method for obtaining the encoding vector as described in step 3-1 includes: For music as input data, the input music spectrum is first obtained through Fourier transform, and then input into a music encoder built by a convolutional neural network (CNN) to obtain a dimensional... The pre-trained music encoding vector, where d is the output dimension of the convolutional neural network CNN, and T is the dimension that varies with the length of the input music; For text data as input data, it is processed into token encoding after word segmentation and input into the text encoder. The text encoder adopts a neural network model to obtain a pre-trained text encoding vector. The output dimension of the text encoder is the same as the output dimension d of the convolutional neural network CNN.
4. The automated music description generation method according to claim 3, characterized in that, The method for obtaining the restoration result as described in step 3-2 includes: The pre-trained music encoding vector is input into the music decoder built by the convolutional neural network (CNN) to restore the music spectrum; the pre-trained text encoding vector is input into the text decoder to restore the text.
5. The automated music description generation method according to claim 4, characterized in that, The specific method for calculating the loss as described in step 3-3 includes: For the music reconstruction result, i.e., the reconstructed music spectrum, sharpening is performed on both the reconstructed and input music spectra, and then the mean square error (MSE) loss is calculated. Specific methods include: set up Given an audio spectrum, where, The frequency domain width represents the spectrum. Indicates the time range of the spectrum; Let the sharpening function be defined as taking a submatrix from row a to row b from the end of the spectrum matrix. for: ； loss function for: ； in, This represents the audio sharpening result obtained by the music codec. For the text reconstruction result, i.e., the reconstructed text, the cross-entropy between the decoded probability distribution in the reconstructed text and the input text data is calculated. Specific methods include: Suppose that the word at position i in the input text data is... The text codec predicts that the probability of this position being word j is . Then the cross-entropy loss for: ； in, This represents a power function with the natural constant e as its base.
6. The automated music description generation method according to claim 5, characterized in that, Steps 3-4 describe the pre-training process, which includes the following specific methods: The loss calculated in step 3-3 Backpropagation is performed on the model parameters of the music description generation model MuTeNet. Specific methods include: assumed These are arbitrary parameters in the music description generation model MuTeNet, used to calculate the loss. For parameters The gradient is Based on the gradient results and the set learning rate Update model parameters: ； Repeat this step to iteratively optimize the model until pre-training is complete.
7. The automated music description generation method according to claim 6, characterized in that, The specific methods for preparing training data as described in step 4 include: Collect music and corresponding comment text, divide the music and corresponding comment text into training set and validation set according to a set ratio, input the training set into the music description generation model for training; input the data in the training set into the model for training in order of text length from shortest to longest.
8. The automated music description generation method according to claim 7, characterized in that, The training described in step 4 includes the following specific methods: Step 4-1: Input the music from the training set into the music encoder to obtain the music encoding vector; Step 4-2: Input the music encoding vector into the text decoder. The text decoding process is the same as in Step 3-2. The text decoder outputs the word probability distribution for each position. Calculate the cross-entropy loss by comparing the generated word probabilities with the corresponding comment text. The text codec converts the descriptive text into text encoding, and the resulting text encoding guides the music encoding training through comparative learning. The specific method is as follows: First, calculate the similarity between different text encodings. ,in It is the encoding result of the i-th sentence in the dataset by the text encoder. This is the encoding result of the j-th sentence in the dataset; then, the contrastive learning loss is calculated based on the similarity between music encodings and the similarity between text encodings. : ； in, This represents the vector encoding obtained by the music encoder for the i-th piece of music in the dataset. This represents the vector encoding of the j-th music in the dataset, where the elements with the same index... and Correspondence, that is It is for music The text description, ; Calculate total loss : ； Step 4-3, calculate the total loss. Backpropagation is performed using the same method as in steps 3-4, and the parameters of the music description generation model are updated to complete the training. Step 4-4: Use the validation set to evaluate the performance of the music description generation model. That is, input the music in the validation set into the music description generation model and compare the similarity between the output description and the comment text corresponding to the music in the validation set. Steps 4-5 involve repeated validation, with the model that yields the best validation result being selected as the final music description generation model.
9. The automated music description generation method according to claim 8, characterized in that, The text encoder described in step 3-1 uses a neural network model that adapts to sequence modeling, namely Transformer or LSTM. The text decoder described in step 3-2 uses a neural network model that adapts to sequence modeling, namely Transformer or LSTM.