Tibetan music melody generation method based on multi-order context frequency domain attention

Abstract

The invention relates to the technical field of Tibetan music, and provides a Tibetan music melody generation method based on multi-order context frequency domain attention, and the method comprises the steps: 1, a coding stage; inputting a Tibetan music melody S, and obtaining feature extraction of the music melody S through a coding module E; step 2, a decoding stage; inputting a result MS after random masking of the music melody S, and obtaining music generation of the randomly masked music melody MS through a decoding module D; 3, a music capsule reasoning stage; the features after the decoding stage pass through a capsule neural network, music sequences form a low-level capsule in a vector mode, key information in the low-level capsule is mapped to a high-level capsule through dynamic routing, the logic relation between sub-sequences is modeled, and music with integrity and logicality is generated through a model. According to the invention, the Tibetan music melody can be better generated.

Description

Technical Field

[0001] The present invention relates to the technical field of Tibetan music, and in particular to a Tibetan music melody generation method based on multi-order context frequency domain attention. Background Art

[0002] Automatic music generation has a history of more than 60 years. During this period, many researchers have adopted a variety of methods, including but not limited to grammatical rules, probabilistic models, evolutionary computation, neural networks, etc., and carried out a lot of research on different data sets and generation tasks. With the rise of deep learning, using deep learning technology to automatically generate various content (such as images, text, etc.) has become a hot research topic. As one of the most common and infectious art forms in human society, music has attracted the attention of many researchers. So far, automatic music generation has derived many subtasks, such as melody generation, chord generation, style transfer, performance generation, audio synthesis, singing synthesis, etc.

[0003] With the development of the field of automatic music generation, many research articles have appeared. RNN is the first neural network model for music generation, and LSTM, as a variant of RNN, is widely used in music generation. There are also some models based on convolutional neural networks. The MusicVAE model is based on Transformer, which learns from a large amount of music data and controls the generated music features by adjusting the parameters of the model. It can generate high-quality and creative music clips, and the model has good scalability. Since piano rolls are non-differentiable discrete data, some scholars proposed a coupling mechanism to learn the information of the latent space and capture the internal and joint distribution of each track, converting the problem into a differentiable optimization problem. Adversarial neural networks have great potential, so some scholars generate music melodies based on GAN, and design generators and discriminators respectively. This method can use prior knowledge to follow the chord sequence to generate melodies. Obviously, Transformer shows great potential in the field of music generation. It can automatically learn the characteristics and patterns of music by learning a large amount of music data, and generate high-quality, creative and diverse music clips with good scalability. Some scholars applied Transformer to create music with long structures for the first time. Due to the huge number of parameters in the Transformer model, some studies introduced transfer learning to optimize model training.

[0004] Most of the above methods focus on learning the basic rules embedded in the music melody structure, but often ignore the local multi-step context features in the music melody and the rich information in the frequency domain. 1) Regarding the local context features, for the expression of Tibetan folk songs, the first few measures of the music melody gradually establish a soothing and elegant atmosphere through delicate note changes and rhythm advancement. If only relying on the overall melody structure and ignoring these local context features, the model may not be able to capture the delicate emotional changes gradually advancing in the melody, and the generated music lacks progressive emotional expression. 2) Regarding the richness of frequency domain information, the changes in the frequency domain are the core of emotional expression. The alternation of the bass in the low-frequency band and the synthesizer in the high-frequency band creates a strong sense of rhythm and space. If these change features in the frequency domain are ignored, the generated music will not be able to present the due level and emotional ups and downs. 3) In addition, Tibetan folk songs have a certain logic. Since Tibetan folk songs usually use unique throat singing methods, such as long tones and trills, and the commonly used instruments are also special, including morin khuur, bili, hand drums, etc., it is an important content to enable the model to generate music with integrity and logic. Summary of the Invention

[0005] The content of the present invention is to provide a method for generating Tibetan music melody based on multi-order context frequency domain attention, which can preferably generate Tibetan music melody.

[0006] A method for generating Tibetan music melody based on multi-order context frequency domain attention according to the present invention includes the following steps:

[0007] Step 1, encoding stage; input the Tibetan music melody S, and obtain the feature extraction of the music melody S through the encoding module E; the encoding module E includes position encoding PE, multi-stage context-aware local attention mechanism MOCA-attention, and spatio-temporal wavelet convolution module STW-CNN;

[0008] Step 2, decoding stage; input the result MS after randomly masking the music melody S, and obtain the music generation of the randomly masked music melody MS through the decoding module D; the decoding module D includes PE, MOCA-attention, STW-CNN, and cross-modal attention mechanism MOCA-cross-attention;

[0009] For the randomly masked music melody MS, obtain the position encoding feature through the PE module, obtain the multi-stage context-aware local attention mechanism feature through MOCA-attention, and obtain the spatio-temporal wavelet convolution feature through STW-CNN;

[0010] Step 3, music capsule reasoning stage; the features after the decoding stage are passed through the capsule neural network, and the music sequence is organized into low-level capsules in the form of vectors. The key information in the low-level capsules is mapped to the high-level capsules through dynamic routing to model the logical relationship between subsequences, so that the model can generate music with both integrity and logic.

[0011] Preferably, in step 1, specifically:

[0012] 1.1) The Tibetan music melody S is passed through the PE module to obtain the sequence x=(x1,x2,x3.....x n ), x is context-encoded for all adjacent keys in the input sequence through a convolution with a kernel size of 1×3 to obtain the encoded local static context key: k = x·W τ , where W τ Represents the parameters of the convolution operation;

[0013] 1.2) Concatenate the obtained local static context key k with the encoded sequence x, and obtain the attention matrix A through two consecutive 1×1 convolution operations; the specific operation is: A = softmax(mean([k,x]·W ω ·W β )),W ω ,W β Represents the parameters of the 1×1 convolution operation;

[0014] 1.3) Calculate v through 1×1 convolution: v = x·W v , v represents the value, the convolution with kernel 1 acts as a fully connected function, W v Represents the convolution parameters; a multi-order context-aware module is introduced to expand the local receptive field. The specific operations of the context-aware module are:

[0015] First, v is compressed to half through the bottleneck structure to reduce the amount of calculation, which is calculated as v'. The specific calculation formula is: v'=v·W α , where W α Parameters representing data compression, i.e., convolution parameters;

[0016] Then, multi-level context is captured by convolutions with different expansion rates, counted as v", and the calculation formula is: v″=v′·W1⊕v′·W2⊕v′·W3⊕v′·W4, where W i ,i={1,2,3,4} represents the convolution parameters with different expansion rates, ⊕ represents the addition operation;

[0017] 1.4) Reshape v' back to its original shape through 1×1 convolution, counted as v'', and its calculation formula is: v'' = v''·W λ , where Wλ Represents the parameters of the 1×1 convolution operation; then, the context-aggregated A and v″′ are matrix multiplied to achieve the weighting of v″′ on A, thereby calculating the new key, i.e. k', and obtaining:

[0018] 1.5) Design a wavelet decomposer with back propagation of all parameters, which can adaptively learn and train all parameters;

[0019] For the input audio sequence x={x1,....x t ,....,x T}, the low-frequency and high-frequency components of the i-th layer obtained by wavelet decomposition are represented as x l (i) and x h (i) For the i-th layer, two sets of filters are introduced to perform convolution operations on the low-frequency subsequences of the i-1th layer, which can be specifically expressed as: and in is a low-frequency subsequence, i represents the number of layers, and n represents the nth segment in the sequence. It is obtained by l (i) and r h (i) obtained by downsampling, r l (i) and r h (i) represents the features obtained by convolution of low-frequency and high-frequency components, respectively. and are respectively learnable mapping parameters; the above operation obtains a high-frequency component and n low-frequency components, and the subsequences output at different levels have different time and frequency resolutions; the frequency resolution increases with the increase of i, while the time resolution decreases continuously, which fully extracts the multi-order frequency features in the music sequence;

[0020] 1.6) The wavelet function is used as a sliding window to perform Hadamard product with the music sequence, which is specifically expressed as: lorh =σ(W lorh (i)x l (i-1)+b lorh (i)), where σ represents the Sigmoid activation function, b is a trainable bias matrix with an initial value distributed around 0, W is a learnable filter coefficient matrix, and the matrix is ​​trained according to the perturbation of the audio data. l orh indicates that the formula can operate on low-frequency and high-frequency components respectively;

[0021] 1.7) Input the high and low frequency components of the wavelet decomposition output into the LSTM module to learn the correlation between melody and frequency in the music sequence;

[0022] 1.8) Stabilize training through residual connections and normalization, and enhance nonlinear representation through multi-layer perceptrons.

[0023] Preferably, in step 1.4), in order to learn the dependency between adjacent subsequences, the input sequence is reshaped by superimposing adjacent subsequences; the relationship between sequences is captured by 3×3 convolution to expand the receptive field of the macro structure as much as possible; j' captures the dynamic feature interaction between inputs and obtains rich dynamic contextual knowledge; the relationship between sequences is captured by 3×3 convolution to expand the receptive field of the macro structure as much as possible, and the feature o is obtained: o=x·W γ , where W γ Represents the weight of the convolution operation; finally, k, k', and o are concatenated and fused through linear projection to obtain the final output: output = [k, k', o] · W υ , where W υ Represents the weight of the linear projection.

[0024] Preferably, in step 1.7), the LSTM module is a multi-level LSTM architecture, and high and low frequency components with different time resolutions are input into multiple sub-LSTM modules, and each sub-LSTM network only predicts each sub-sequence of the music sequence; finally, the music sequences output by each sub-LSTM network are fused through a convolution module to obtain the final output.

[0025] Preferably, in step 3, specifically:

[0026] First, the output of the previous module is mixed through the fully connected layer, expressed as: x1 = x W fc1 , where x1 is the mixed sequence, W fc1 is a learnable parameter; secondly, a capsule network is designed, where the initial capsule is the first layer of the low-level capsule. Each capsule in the capsule network can represent a vector of multiple features of the music sequence, and the corresponding weight parameter is also a vector; the input s of the j-th layer capsule is j It is the vector after integrating the initial information. After iteration, the parameters are passed to get the output v j , the lower-layer capsules are linearly weighted summed, the weights representing the bottom-layer capsules are calculated, and the next-layer capsules are predicted; the input of the capsule network of the j-th layer is defined as s j , specifically expressed as: Among them, w ij is the weight matrix to be multiplied by each mapping capsule vector, u i represents the initial capsule of the i-th layer; the calculation process of the dynamic routing algorithm is regarded as each capsule neuron in the previous layer outputs to a neuron in the next layer with different strengths of connection; based on the above operations, the output v can be obtained j :v j =[||sj || 2 / (1+||s j || 2 )]·(s j / ||s j ||); The above operation is used to introduce sparse weights of high-level capsules to correct the connection strength between low-level capsules and high-level capsules in the next iteration, thereby optimizing the dynamic routing algorithm.

[0027] As a preference, the similarity between the input and output of the capsule network is checked by dot product operation, and the similarity between the input and output of the capsule network is checked by iterating w ij , that is, determine a corresponding path between capsules. The modules of the capsule neurons on this path are large, so the probability of the low-level capsule being selected by the high-level capsule is large, and the end of the path is the correctly predicted capsule; finally, the parent layer capsule vector is obtained and recast into a one-dimensional music sequence to obtain the final predicted music sequence.

[0028] As a preference, the loss function during the entire training process is: where y predict Represents the predicted result, y real represents the true result, and t represents the position of the music sequence.

[0029] The beneficial effects of the present invention are as follows:

[0030] The present invention designs a multi-stage context-aware attention mechanism to implement multi-scale convolution to fully extract the local context features of the music sequence, and associates the key with the original sequence to capture the dynamic substructure information of the sequence; designs a multi-level wavelet decomposition to subdivide the frequency domain features in folk songs step by step, and extracts the long-term dependencies of different frequency domain components through LSTM to learn the unique and melodious melody in folk music. In order to improve the logic of music generation, the Transformer model structure of the capsule neural network is designed to encode the music sequence to form a low-level capsule, and map the key information in the low-level capsule to the high-level capsule through dynamic routing to model the logical relationship between subsequences, so that the model generates music with both integrity and logic. This method shows stronger expressive power when modeling music sequences, and effectively makes up for the problem of insufficient utilization of local features and frequency domain information in existing methods.

[0031] The present invention has the following advantages:

[0032] 1) Enhanced emotional expression: By capturing local multi-step context features, the model can more carefully restore the gradual emotional changes in music, making the generated music more in line with human auditory experience. For example, the model can achieve a more natural transition between the melody and the climax, enhancing the appeal of the music.

[0033] 2) Improve the coherence and layering of the melody: The introduction of frequency domain information allows the model to not only focus on the changes in the note sequence in the time domain, but also use the features in different frequency bands to enrich the layering of the music. In this way, the generated music can better express complex harmonic structures and rhythm changes, presenting a richer musical texture.

[0034] 3) Improve the generalization ability of the model: Traditional methods tend to generate music clips with high repetitiveness or a single style because they ignore local features and frequency domain information. By combining local context and frequency domain features, this method can generate music clips with higher diversity and richer styles, effectively improving the generalization ability of the model in different styles and emotional scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 A schematic diagram of a Tibetan music melody generation method based on multi-order contextual frequency domain attention;

[0036] Figure 2 Schematic diagram of the multi-stage context-aware local attention mechanism MOCA-attention;

[0037] Figure 3 Schematic diagram of the spatiotemporal wavelet convolution module STW-CNN;

[0038] Figure 4 Schematic diagram of the music reasoning module based on capsule neural network;

[0039] Figure 5 Generate comparative diagrams for Tibetan music waveforms and spectra; Figure 6 This is a diagram for visualizing the acoustic spectrum; Figure 7 Schematic diagram for evaluating generated music in different dimensions. DETAILED DESCRIPTION

[0040] In order to further understand the content of the present invention, the present invention is described in detail in conjunction with the accompanying drawings and embodiments. It should be understood that the embodiments are only for explaining the present invention and are not intended to limit it.

[0041] like Figure 1 As shown, this embodiment provides a Tibetan music melody generation method based on multi-order context frequency domain attention, which includes the following steps:

[0042] Step 1, encoding stage; input Tibetan music melody S, and obtain the feature extraction of music melody S through encoding module E; encoding module E includes position encoding PE, multi-stage context-aware local attention mechanism MOCA-attention and spatiotemporal wavelet convolution module STW-CNN.

[0043] Step 2, decoding stage; input the result MS after randomly masking the music melody S, and obtain the music generation of the randomly masked music melody MS through the decoding module D; the decoding module D includes PE, MOCA-attention, STW-CNN and cross-modal attention mechanism MOCA-cross-attention.

[0044] For the randomly masked music melody MS, the PE module is used to obtain the position encoding features, and the multi-stage context-aware local attention mechanism features are obtained through MOCA-attention, and the spatiotemporal wavelet convolution features are obtained through STW-CNN.

[0045] Step 3, music capsule reasoning stage; the features after the decoding stage are passed through the capsule neural network, and the music sequence is organized into low-level capsules in the form of vectors. The key information in the low-level capsules is mapped to the high-level capsules through dynamic routing to model the logical relationship between subsequences, so that the model can generate music with both integrity and logic.

[0046] In step 1, specifically:

[0047] 1.1) The Tibetan music melody S is passed through the PE module to obtain the sequence x=(x1,x2,x3.....x n ), x is context-encoded for all adjacent keys in the input sequence through a convolution with a kernel size of 1×3 to obtain the encoded local static context key: k = x·W τ , where W τ Represents the parameters of the convolution operation.

[0048] 1.2) Concatenate the obtained local static context key k with the encoded sequence x, and obtain the attention matrix A through two consecutive 1×1 convolution operations; the specific operation is: A = soft max(mean([k,x]·W ω ·W β )),W ω ,W β Represents the parameters of a 1×1 convolution operation.

[0049] 1.3) Calculate v through 1×1 convolution: v = x·W v , v represents the value, the convolution with kernel 1 acts as a fully connected function, W v Represents the convolution parameters; a multi-order context-aware module is introduced to expand the local receptive field. The specific operations of the context-aware module are:

[0050] First, v is compressed to half through the bottleneck structure to reduce the amount of calculation, which is calculated as v'. The specific calculation formula is: v'=v·W α , where Wα Parameters representing data compression, i.e., convolution parameters;

[0051] Then, multi-level context is captured by convolutions with different expansion rates, counted as v", and the calculation formula is: v″=v′·W1⊕v′·W2⊕v′·W3⊕v′·W4, where W i ,i={1,2,3,4} represents the convolution parameters with different expansion rates, and ⊕ represents the addition operation.

[0052] 1.4) Reshape v' back to its original shape through 1×1 convolution, counted as v'', and its calculation formula is: v'' = v''·W λ , where W λ Represents the parameters of the 1×1 convolution operation; then, the context-aggregated A and v″′ are matrix multiplied to achieve the weighting of v″′ on A, thereby calculating the new key, i.e. k', and obtaining:

[0053] In order to learn the dependencies between adjacent subsequences, we choose to reshape the input sequence by superimposing adjacent subsequences; the relationship between sequences is captured by 3×3 convolution to maximize the receptive field of the macro structure; k' captures the dynamic feature interaction between inputs and obtains rich dynamic contextual knowledge; the relationship between sequences is captured by 3×3 convolution to maximize the receptive field of the macro structure, and feature o is obtained: o = x·W γ , where W γ Represents the weight of the convolution operation; finally, k, k', and o are concatenated and fused through linear projection to obtain the final output: output = [k, k', o] · W υ , where W υ Represents the weight of the linear projection.

[0054] 1.5) Design a wavelet decomposer with back propagation of all parameters, which can adaptively learn and train all parameters.

[0055] For the input audio sequence x={x1,....x t ,....,x T}, the low-frequency and high-frequency components of the i-th layer obtained by wavelet decomposition are represented as x l (i) and x h (i) For the i-th layer, two sets of filters are introduced to perform convolution operations on the low-frequency subsequences of the i-1th layer, which can be specifically expressed as: and in is a low-frequency subsequence, i represents the number of layers, and n represents the nth segment in the sequence. It is obtained by l (i) and rh (i) obtained by downsampling, r l (i) and r h (i) represents the features obtained by convolution of low-frequency and high-frequency components, respectively. and are respectively learnable mapping parameters; the above operation obtains a high-frequency component and n low-frequency components, and the subsequences output at different levels have different time and frequency resolutions; the frequency resolution increases with i, while the time resolution decreases continuously, which fully extracts the multi-order frequency features in the music sequence.

[0056] 1.6) The wavelet function is used as a sliding window to perform Hadamard product with the music sequence, which is specifically expressed as: lorh =σ(W lorh (i)x l (i-1)+b lorh (i)), where σ represents the Sigmoid activation function, b is a trainable bias matrix with an initial value distributed around 0, W is a learnable filter coefficient matrix, and the matrix is ​​trained according to the perturbation of the audio data. l orh indicates that the formula can operate on low-frequency and high-frequency components respectively.

[0057] 1.7) The high and low frequency components of the wavelet decomposition output are input into the LSTM module to learn the correlation between melody and frequency in the music sequence.

[0058] The LSTM module is a multi-level LSTM architecture. High and low frequency components with different time resolutions are input into multiple sub-LSTM modules. Each sub-LSTM network only predicts each sub-sequence of the music sequence. Finally, a convolution module is used to merge the music sequences output by each sub-LSTM network to obtain the final output.

[0059] 1.8) Stabilize training through residual connections and normalization, and enhance nonlinear representation through multi-layer perceptrons.

[0060] In step 3, specifically:

[0061] First, the output of the previous module is mixed through the fully connected layer, expressed as: x1 = x W fc1 , where x1 is the mixed sequence, W fc1 is a learnable parameter; secondly, a capsule network is designed, where the initial capsule is the first layer of the low-level capsule. Each capsule in the capsule network can represent a vector of multiple features of the music sequence, and the corresponding weight parameter is also a vector; the input s of the j-th layer capsule is j It is the vector after integrating the initial information. After iteration, the parameters are passed to get the output v j, the lower-layer capsules are linearly weighted summed, the weights representing the bottom-layer capsules are calculated, and the next-layer capsules are predicted; the input of the capsule network of the j-th layer is defined as s j , specifically expressed as: s j =∑ i w ij u i , where w ij is the weight matrix to be multiplied by each mapping capsule vector, u i represents the initial capsule of the i-th layer; the calculation process of the dynamic routing algorithm is regarded as each capsule neuron in the previous layer outputs to a neuron in the next layer with different strengths of connection; based on the above operations, the output v can be obtained j :v j =[||s j || 2 / (1+||s j || 2 )]·(s j / ||s j ||); The above operation is used to introduce sparse weights of high-level capsules to correct the connection strength between low-level capsules and high-level capsules in the next iteration, thereby optimizing the dynamic routing algorithm.

[0062] The similarity between the input and output of the capsule network is checked by dot product operation, and the similarity between the input and output of the capsule network is checked by iterating w ij , that is, determine a corresponding path between capsules. The modules of the capsule neurons on this path are large, so the probability of the low-level capsule being selected by the high-level capsule is large, and the end of the path is the correctly predicted capsule; finally, the parent layer capsule vector is obtained and recast into a one-dimensional music sequence to obtain the final predicted music sequence.

[0063] The loss function during the entire training process is: where y predict Represents the predicted result, y real represents the true result, and t represents the position of the music sequence.

[0064] This embodiment has the following characteristics:

[0065] Attention mechanism of multi-order context aggregation: The traditional attention mechanism can only capture global dependencies and cannot refine and extract the unique structure in Tibetan folk songs. However, the music sequence contains rich context information. The conventional attention mechanism only learns the relationship between paired queries and keys, ignoring the dynamic context between neighborhood keys. At the same time, it only learns the music structure in the time dimension, ignoring the causal relationship between different subsequences. For this reason, this embodiment proposes an attention mechanism of multi-order context aggregation, which aims to fully perceive the context information contained in the music and learn and supplement the dependencies between different sub-regions. In order to better weigh the importance of a certain point in the sequence, this embodiment chooses to introduce a multi-order context perception module to expand the local receptive field. The specific operation of context perception is to first compress the data to half through a bottleneck structure to reduce calculations, and then capture multi-order contexts through convolutions with different expansion rates. Then, the context-aggregated A and value are matrix multiplied to achieve the weighting of value to A to capture the dynamic feature interactions between the inputs and obtain rich dynamic context knowledge. As Figure 2 As shown, it is a multi-stage context-aware local attention mechanism MOCA-attention.

[0066] Spatiotemporal wavelet convolution module: The Transformer model lacks effective modeling of frequency information in the analysis of music sequences. This embodiment proposes a multi-level wavelet decomposition network structure based on wavelet transform, which is used to analyze and perceive frequency information in music sequences, generate audio data with frequency logic according to hidden trend reasoning at different frequencies, and finally extract local features through convolution to adaptively fuse high and low frequency components, fully extract effective frequency domain information, and learn the characteristics of folk music frequency. In order to solve the problem that the sparse coupling method of wavelet decomposition does not use independent parameters for global optimization and is difficult to integrate into the Transformer architecture, this embodiment designs a wavelet decomposer in which all parameters can be back-propagated, which can adaptively learn and train all parameter hungry solutions. That is, this module can not only use wavelet decomposition to extract features from time series, but also use back-propagation to update all parameters in the training stage. Specifically, multi-level discrete wavelet decomposition decomposes the time series into low-frequency and high-frequency sub-sequences step by step, thereby extracting multi-order frequency features. For the input audio sequence, the low-frequency and high-frequency components of the i-th layer are obtained by wavelet decomposition. For the i+1th layer, two sets of filters are introduced to perform convolution operations on the low-frequency subsequences of the i layer. This operation obtains a high-frequency component and n low-frequency components. The output subsequences of different levels have different time and frequency resolutions. The frequency resolution increases as i increases, while the time resolution decreases continuously, fully extracting the multi-order frequency features in the music sequence. Next, in order to embed the wavelet decomposition into the deep learning model structure, the filter coefficients in the wavelet transform function become learnable parameters. The specific approach is to make the wavelet function do the Hadamard product with the music sequence in the form of a sliding window. Then, the high and low frequency components output by the wavelet decomposition are input into the LSTM module to learn the correlation between melody and frequency in the music sequence. A multi-level LSTM architecture is used here, and the high and low frequency components with different time resolutions are input into multiple sub-LSTM modules. This is because the high and low frequency components of the subsequence are relatively simple. The climax or exciting part of the music is usually at high frequency, while the melody is usually at low frequency. Each sub-LSTM network only predicts each sub-music sequence. Finally, the music sequences output by each sub-LSTM network are fused through a convolution module to obtain the final output result. As shown Figure 3 As shown, it is the spatiotemporal wavelet convolution module STW-CNN.

[0067] Music reasoning module based on capsule neural network: The model often cannot understand the hidden logic of the music melody in the entire sequence, ignores the relationship between substructures in the macro structure, is insensitive to spatial information, is difficult to encode effectively, and lacks the ability to express music. To this end, the capsule neural network is introduced to encapsulate the important information of the state in the music sequence features into the capsule in the form of a vector, automatically model the relationship between the local and the whole, and make the generated audio interpretable. Considering that convolution can retain the characteristics of the music sequence in the music sequence processing task, the features of the sequence are extracted through causal expansion convolution to increase the receptive field. First, the output of the previous module is mixed through the FC layer, and feature extraction is performed through two layers of standard convolution layers to learn the multi-angle features of the music sequence. Then, the features are converted into initial capsules u, and the data information of the lower layer features is read through the capsule convolution layer, and the respective activity vectors are output. Each layer of initial capsules u contains attribute features such as semantics, location, and even emotion. The input u of the initial capsule of the nth layer is a vector after integrating the primary information. After iteration, the parameters are passed to the n+1 layer to obtain the output v. The lower-level capsules are linearly weighted summed to calculate the weight c representing the bottom-level capsule and predict the next-level capsule. The weights of the lower-level capsules are processed by the ReLU activation function to ensure that they have nonlinear expression capabilities: c ij =b ij ·λ j ,λ j =||v j || is used to introduce sparse weights of high-level capsules to correct the connection strength between low-level capsules and high-level capsules in the next iteration, thereby optimizing the dynamic routing algorithm. The similarity between the input and output of the capsule network is checked by dot product operation, and b is updated by iteration. ij Determine c ij , that is, determine a corresponding path between capsules. The module of the capsule neurons on this path is large, so the probability of the low-level capsule being selected by the high-level capsule is large, and the end of the path is the correctly predicted capsule. Finally, the parent layer capsule vector is obtained, where s is the number of subsequences. The two-dimensional capsule matrix V will be recast into a one-dimensional music sequence, and finally contact the fully connected layer to map it into a vector of length l, which is the final predicted music sequence. Figure 4 As shown, it is a music reasoning module based on capsule neural network.

[0068] Experimental analysis

[0069] Experimental data set: This method uses a self-collected data set of 3,600 Tibetan folk songs. MIDI data needs to be standardized before inputting into the neural network. Each input is 8 bars of music, and the beat is standardized to 4 / 4 beats, and the minimum note unit is sixteenth notes. All MIDI files are converted into 2D real-valued vectors. The data set is divided into 90% as a training set and 10% as a test set.

[0070] Experimental settings: This method is suitable for experiments and tests in the PyTorch framework. The number of iterations is selected as 230, the loss rate is 0.1, and the batch size is 32. The model hyperparameters are set to depth 12, time embedding length 512, and learning rate 0.002. The entire experiment is conducted on a server with an AMD 5900X CPU, 4 RTX 8000 GPUs, and 128GB of memory.

[0071] Experimental results:

[0072] 1. Tibetan music generation index evaluation: In order to accurately evaluate the advantages of the model performance, this embodiment uses three indicators: the accuracy of the generated music sequence (p), the root mean square error (RMSE) and the mean square error (MSE) between the generated sequence and the true sequence to evaluate its performance, as shown in Table 1, where GRU, BIGRU, Midinet, Autoformer, MTS-Mixer, and Music Transformer are all existing models.

[0073] Table 1 Generation index evaluation table

[0074] Model P MSE RMSE GRU 0.811 0.271 0.415 BIGRU 0.852 0.173 0.384 Midinet 0.883 0.134 0.243 Autoformer 0.910 0.065 0.229 MTS-Mixer 0.917 0.050 0.224 MusicTransformer 0.904 0.082 0.212 Ours 0.935 0.051 0.152

[0075] The pitch features of the model are evaluated, including pitch variation (PV), qualified note ratio (QN), and the average number of pitch types used per measure (UPC), as shown in Table 2.

[0076] Table 1 Pitch feature evaluation table

[0077]

[0078]

[0079] 2. Comparison of Tibetan music waveform and spectrum generation: Figure 5 As shown in the figure. The left side shows the original music, and the right side shows the generated music. It can be seen that the waveforms of the original music and the generated music are similar and the diversity is consistent.

[0080] 3. Visualization of acoustic spectrum. Figure 6 As shown. The spectrogram is three-dimensional, where the x-axis represents time, the y-axis represents frequency, and the z-axis is the signal strength corresponding to a specific time and frequency, represented by the depth of the color. As can be seen from the figure below, the low-frequency part of the melody generated by the model is responsible for the rhythm, while the high-frequency part is responsible for the melody. These characteristics are consistent with the characteristics of folk songs.

[0081] 4. Human evaluation of Tibetan music generation: The generated music is evaluated in batches in different dimensions, including pleasure, fluency, and naturalness. The generated Tibetan music is compared with the real Tibetan music, and the average score of 10 professionals who have learned Tibetan music theory is taken to get the score of each song, with a total score of 10 points. Figure 7 shown.

[0082] The present invention and its embodiments are described schematically above, and the description is not restrictive. The drawings show only one embodiment of the present invention, and the actual structure is not limited thereto. Therefore, if a person skilled in the art is inspired by it and designs a structural method and an embodiment similar to the technical solution without creativity without departing from the purpose of the invention, they shall all fall within the protection scope of the present invention.

Claims

1. A Tibetan music melody generation method based on multi-order contextual frequency domain attention, characterized by: The following steps are involved: Step 1, encoding stage: input Tibetan music melody S, and extract the features of the music melody S through the encoding module E; the encoding module E includes position encoding PE, multi-stage context-aware local attention mechanism MOCA-attention and spatiotemporal wavelet convolution module STW-CNN; Step 2, decoding stage; input the result MS after random masking of the music melody S, and obtain the music generation of the random masked music melody MS through the decoding module D; the decoding module D includes PE, MOCA-attention, STW-CNN and cross-modal attention mechanism MOCA-cross-attention; For the randomly masked music melody MS, the PE module is used to obtain the position encoding feature, and the multi-stage context-aware local attention mechanism feature is obtained through MOCA-attention, and the spatiotemporal wavelet convolution feature is obtained through STW-CNN; Step 3: Music capsule reasoning stage; The features after the decoding stage are passed through the capsule neural network to organize the music sequence into low-level capsules in the form of vectors, and the key information in the low-level capsules is mapped to the high-level capsules through dynamic routing to model the logical relationship between sub-sequences, so that the model can generate music with both integrity and logic.

2. The Tibetan music melody generation method based on multi-order contextual frequency domain attention according to claim 1 is characterized in that: In step 1, specifically: 1.1) The Tibetan music melody S is passed through the PE module to obtain the sequence x=(x1,x2,x3.....x n ), x is context-encoded for all adjacent keys in the input sequence through a convolution with a kernel size of 1×3 to obtain the encoded local static context key: k = x·W τ , where W τ Represents the parameters of the convolution operation; 1.2) Concatenate the obtained local static context key k with the encoded sequence x, and obtain the attention matrix A through two consecutive 1×1 convolution operations; the specific operation is: A = softmax(mean([k,x]·W ω ·W β )),W ω ,W β Represents the parameters of the 1×1 convolution operation; 1.3) Calculate v through 1×1 convolution: v = x·W v , v represents the value, the convolution with kernel 1 acts as a fully connected function, W v Represents the convolution parameters; a multi-order context-aware module is introduced to expand the local receptive field. The specific operations of the context-aware module are: First, v is compressed to half through the bottleneck structure to reduce the amount of calculation, which is calculated as v'. The specific calculation formula is: v'=v·W α , where W α Represents the parameters of data compression, i.e., convolution parameters; Then, multi-level contexts are captured by convolutions with different expansion rates, counted as v", and the calculation formula is: Where W i ,i={1,2,3,4} represents the convolution parameters with different expansion rates, Represents addition operation; 1.4) Reshape v' back to its original shape through 1×1 convolution, counted as v'', and its calculation formula is: v'' = v''·W λ , where W λ Represents the parameters of the 1×1 convolution operation; then, the context-aggregated A and v″′ are matrix multiplied to achieve the weighting of v″′ on A, thereby calculating the new key, i.e. k', and obtaining: 1.5) Design a wavelet decomposer with back propagation of all parameters, which can adaptively learn and train all parameters; For the input audio sequence x={x1,....x t ,....,x T }, the low-frequency and high-frequency components of the i-th layer obtained by wavelet decomposition are represented as x l (i) and x h (i) For the i-th layer, two sets of filters are introduced to perform convolution operations on the low-frequency subsequences of the i-1th layer, which can be specifically expressed as: and in is a low-frequency subsequence, i represents the number of layers, and n represents the nth segment in the sequence. It is obtained by l (i) and r h (i) obtained by downsampling, r l (i) and r h (i) represents the features obtained by convolution of low-frequency and high-frequency components, respectively. and are respectively learnable mapping parameters; the above operation obtains a high-frequency component and n low-frequency components, and the subsequences output at different levels have different time and frequency resolutions; the frequency resolution increases with the increase of i, while the time resolution decreases continuously, which fully extracts the multi-order frequency features in the music sequence; 1.6) The wavelet function is used as a sliding window to perform Hadamard product with the music sequence, which is specifically expressed as: lorh =σ(W lorh (i)x l (i-1)+b lorh (i)), where σ represents the Sigmoid activation function, b is a trainable bias matrix with an initial value distributed around 0, W is a learnable filter coefficient matrix, and the matrix is ​​trained according to the perturbation of the audio data. l or h indicates that the formula can operate on low-frequency and high-frequency components respectively; 1.7) Input the high and low frequency components of the wavelet decomposition output into the LSTM module to learn the correlation between melody and frequency in the music sequence; 1.8) Stabilize training through residual connections and normalization, and enhance nonlinear representation through multi-layer perceptrons.

3. The Tibetan music melody generation method based on multi-order contextual frequency domain attention according to claim 2 is characterized in that: In step 1.4), in order to learn the dependency between adjacent subsequences, we choose to reshape the input sequence by superimposing adjacent subsequences; the relationship between sequences is captured by 3×3 convolution to maximize the receptive field of the macro structure; k' captures the dynamic feature interaction between inputs and obtains rich dynamic contextual knowledge; the relationship between sequences is captured by 3×3 convolution to maximize the receptive field of the macro structure, and feature o is obtained: o = x·W γ , where W γ Represents the weight of the convolution operation; finally, k, k', and o are concatenated and fused through linear projection to obtain the final output: output = [k, k', o] · W υ , where W υ Represents the weight of the linear projection.

4. The Tibetan music melody generation method based on multi-order contextual frequency domain attention according to claim 3 is characterized in that: In step 1.7), the LSTM module is a multi-level LSTM architecture. High and low frequency components with different time resolutions are input into multiple sub-LSTM modules. Each sub-LSTM network only predicts each sub-sequence of the music sequence. Finally, a convolution module is used to merge the music sequences output by each sub-LSTM network to obtain the final output.

5. The Tibetan music melody generation method based on multi-order contextual frequency domain attention according to claim 4 is characterized in that: In step 3, specifically: First, the output of the previous module is mixed through the fully connected layer, expressed as: x1 = x W fc1 , where x1 is the mixed sequence, W fc1 is a learnable parameter; secondly, a capsule network is designed, where the initial capsule is the first layer of the low-level capsule. Each capsule in the capsule network can represent a vector of multiple features of the music sequence, and the corresponding weight parameter is also a vector; the input s of the j-th layer capsule is j It is the vector after integrating the initial information. After iteration, the parameters are passed to get the output v j , the lower-layer capsules are linearly weighted summed, the weights representing the bottom-layer capsules are calculated, and the next-layer capsules are predicted; the input of the capsule network of the j-th layer is defined as s j , specifically expressed as: s j =∑ i w ij u i , where w ij is the weight matrix to be multiplied by each mapping capsule vector, u i represents the initial capsule of the i-th layer; the calculation process of the dynamic routing algorithm is regarded as each capsule neuron in the previous layer outputs to a neuron in the next layer with different strengths of connection; based on the above operations, the output v can be obtained j :v j =[||s j || 2 / (1+||s j || 2 )]·(s j / ||s j ||); The above operation is used to introduce sparse weights of high-level capsules to correct the connection strength between low-level capsules and high-level capsules in the next iteration, thereby optimizing the dynamic routing algorithm.

6. The Tibetan music melody generation method based on multi-order contextual frequency domain attention according to claim 5 is characterized in that: The similarity between the input and output of the capsule network is checked by dot product operation, and the similarity between the input and output of the capsule network is checked by iterating w ij , that is, determine a corresponding path between capsules. The modules of the capsule neurons on this path are large, so the probability of the low-level capsule being selected by the high-level capsule is large, and the end of the path is the correctly predicted capsule; finally, the parent layer capsule vector is obtained and recast into a one-dimensional music sequence to obtain the final predicted music sequence.

7. The method for generating Tibetan music melody based on multi-order contextual frequency domain attention according to claim 6, characterized in that: The loss function during the entire training process is: where y predict Represents the predicted result, y real represents the true result, and t represents the position of the music sequence.

Images