A Cantonese opera singing classification method based on CNN and LSTM
Through a multi-layer feature cascade convolutional neural network based on CNN and LSTM, multi-layer features of Cantonese opera audio signals are extracted, which solves the problem that it is difficult for the existing technology to accurately classify Cantonese opera singing styles, and realizes the precise classification and feature extraction of Cantonese opera singing styles. Comprehensive expression and meet practical application needs.
Patent Information
- Application Number
- CN202111313774.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-11-08
- Publication Date
- 2024-12-31
- Estimated Expiration
- 2041-11-08
AI Technical Summary
The existing music classification methods are difficult to accurately classify the rhythm, accompaniment and musical characteristics of different singing styles of Cantonese opera, and it is difficult to extract the deep and shallow features of multi-layer cascades, resulting in the lack of information.
Using a multi-layer feature cascaded convolutional neural network based on CNN and LSTM, multi-level features of audio signals are extracted through the primary network Inception-CNN, and input them into the secondary network CNN-double-layer LSTM to learn the audio signals. Context-related semantics realize the precise classification of Cantonese opera singing styles.
It has achieved comprehensive expression and precise classification of Cantonese opera singing styles, fully extracted the characteristics of Cantonese opera, met the needs of actual application, and improved the accuracy of classification.
Smart Images
Figure CN114067788B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of Cantonese opera singing classification, and in particular to a method for evaluating regional water conservation capacity by adopting a water reserve balance method based on GRACE gravity satellite and remote sensing image data. Background Art
[0002] As a world-class intangible cultural heritage, Cantonese opera has a profound traditional cultural heritage. Making full use of the advantages of artificial intelligence technology to rescue, excavate, organize, protect and spread traditional culture has become the direction of intangible cultural heritage protection today. Therefore, it is of far-reaching significance to achieve the scientific continuation of Chinese traditional culture through the combination of modern science and technology and traditional cultural industries.
[0003] How to process, organize, analyze and utilize massive amounts of audio information is an important part of the information processing field, and audio classification is one of the key technologies. In recent years, research on audio / music style classification based on deep learning has made breakthrough progress. Considering that Cantonese opera singing classification is a branch of the field of music style classification, the research on Cantonese opera singing feature analysis can draw on existing music style classification technology, and on this basis, further consider the characteristics of Cantonese opera singing itself that are different from music, and then put it into practice.
[0004] In recent years, music style classification has continued to grow into a thriving research field. At the same time, more and more activities in this research field are driven by scientific challenges and industrial needs. The current classification of opera singing still has the following technical problems:
[0005] Since the rhythm, accompaniment and musical instruments of different Cantonese opera singing styles are similar, it is difficult to accurately classify them using existing music classification methods.
[0006] Existing music style / audio classification technology can only extract shallow features within Cantonese opera segments. The correlation information between different segments of the same piece has not yet been analyzed and studied. At the same time, it is also difficult to extract multi-layer cascade deep and shallow features based on the characteristics of Cantonese opera singing, and there is a certain possibility of information missing.
[0007] Therefore, how to provide a Cantonese opera singing classification method based on the combination of CNN and LSTM that is comprehensive in expression and accurate in classification is an urgent problem that technicians in this field need to solve. Summary of the invention
[0008] In view of this, the present invention aims to solve the problem that the musical characteristics of different singing styles of Cantonese opera are similar and traditional music classification methods are difficult to accurately classify them. A Cantonese opera singing style classification method based on a multi-layer feature cascade convolutional neural network (CNN) and a long short-term memory unit (LSTM) is proposed.
[0009] In order to achieve the above object, the present invention adopts the following technical solution:
[0010] A Cantonese opera singing classification method based on the combination of CNN and LSTM includes the following steps:
[0011] Steps to build a classification network model:
[0012] Set the cascaded convolutional neural network Inception-CNN as the first-level network, set the CNN-two-layer LSTM neural network as the second-level network, and connect the output of the first-level network to the input of the second-level network;
[0013] Steps to classify Cantonese opera singing using the classification network model:
[0014] Through feature engineering, the Mel spectrum MFCC features are extracted from the Mel spectrum of the Cantonese opera audio signal of each segment;
[0015] The Mel spectrum MFCC features are input into the primary network to fuse the shallow and deep features of the Cantonese opera audio signals of each segment, and the output signal is input into the secondary network;
[0016] The secondary network learns the intrinsic features between the audio signals of each Cantonese opera segment, extracts context-related semantics, and predicts the Cantonese opera singing category.
[0017] Preferably, the primary network includes a two-dimensional convolutional CRMD block and a multi-scale feature extractor;
[0018] The CRMD block of two-dimensional convolution converts the audio signal from the time domain to the frequency domain and extracts multi-level features from both the time and frequency directions;
[0019] The multi-scale feature extractor uses a lightweight Inception structure to split multi-level features through a symmetrical double-layer stacked convolution kernel structure to extract features at different levels;
[0020] The Concat layer connects the features of each level and inputs them into the secondary network.
[0021] Preferably, the secondary network includes a feature cascade stacking network formed by stacking a double-layer LSTM and a multi-convolutional layer CNN. The double-layer LSTM stacking network is used to extract deep separable features layer by layer, and each layer outputs a feature vector as the input of the subsequent layer.
[0022] Preferably, the step of using the classification network model to classify Cantonese opera singing also includes the step of preprocessing the Cantonese opera audio data sample set to obtain a Mel spectrum, including:
[0023] Slice the Cantonese opera audio data sample set into multiple audio files of equal length;
[0024] Performing a frame operation on the audio file, and performing a windowing operation on every two adjacent frame signals using a preset overlap rate;
[0025] Use fast Fourier transform to perform fast Fourier transform on each frame signal to obtain a spectrum diagram;
[0026] The spectrogram is processed by using a Mel filter bank to obtain a Mel spectrum.
[0027] Preferably, a confusion matrix is used to visually judge the prediction results output by the classification network model, and the Cantonese opera singing category is predicted by a majority voting algorithm.
[0028] It can be seen from the above technical solution that, compared with the prior art, the beneficial effects of the present invention include:
[0029] The present invention uses the singing music spectrum corresponding to the Cantonese opera singing segment as the model input, and obtains the shallow and deep fusion feature expressions of each Cantonese opera singing segment through the first-level network (Inception-CNN) composed of 2d-CRMD block and multi-scale feature extractor. Then, through the second-level network (CNN-2LSTM) composed of CNN and stacked LSTM, the contextual association semantics between each singing segment and the importance of each singing segment are learned. Compared with directly borrowing from the existing music style and audio classification algorithms, the Cantonese opera singing classification method based on the combination of multi-layer feature cascade convolutional neural network (CNN) and long short-term memory unit (LSTM) proposed in the present invention is more in line with the characteristics of Cantonese opera singing, fully extracts the characteristics of Cantonese opera, achieves the effect of accurate classification, and has a comprehensive expression, which is more in line with the actual application needs. BRIEF DESCRIPTION OF THE DRAWINGS
[0030] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art are briefly introduced below. Obviously, the drawings in the following description are only embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on the provided drawings without creative work.
[0031] Figure 1 A flow chart of the Cantonese opera singing classification method based on the combination of CNN and LSTM provided by the present invention;
[0032] Figure 2 A schematic diagram of framing an audio signal provided by the present invention;
[0033] Figure 3 A mapping diagram of the audio signal from frequency scale to Mel scale provided by the present invention;
[0034] Figure 4A spectrum diagram of an audio signal provided by the present invention;
[0035] Figure 5 A diagram of the acquisition process of the Mel-spectrogram provided by the present invention;
[0036] Figure 6 A Mel-spectrogram of the audio signal provided by the present invention;
[0037] Figure 7 A network structure diagram of the classification network model provided by the present invention;
[0038] Figure 8 A curve diagram showing the change in the training set accuracy and the test set accuracy of the classification network model provided by the present invention;
[0039] Fig. 9 This is a curve diagram showing the changes in the training set error and the test set error of the classification network model provided by the present invention. DETAILED DESCRIPTION
[0040] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0041] The current mainstream music classification technology is mainly aimed at the style characteristics of music. Cantonese opera as a form of music can learn from this method, but it is more necessary to combine the singing characteristics of Cantonese opera itself to extract multi-dimensional high-level features. See the attached Figure 1 The flowchart of the Cantonese opera singing classification method based on the combination of CNN and LSTM disclosed in this embodiment includes the following steps:
[0042] S1. Extract the Mel spectrum MFCC features from the Mel spectrum of the Cantonese opera audio signal of each segment through feature engineering.
[0043] S2. Input the Mel spectrum MFCC features into the primary network to fuse the shallow and deep features of the Cantonese opera audio signals of each segment, and input the output signal into the secondary network.
[0044] S3. The secondary network learns the intrinsic features between the audio signals of each Cantonese opera segment, extracts context-related semantics, and predicts the Cantonese opera singing category.
[0045] In this embodiment, Convolutional Neural Networks (CNN) is a type of feedforward neural network that includes convolution calculations and has a deep structure, and is one of the representative algorithms of deep learning. Convolutional neural networks have representation learning capabilities and can perform shift-invariant classification on input information according to their hierarchical structure. Both supervised learning and unsupervised learning can be performed, and the convolution kernel parameter sharing in the hidden layer and the sparsity of the inter-layer connection enable the convolutional neural network to perform grid-like topology features with a smaller amount of computation.
[0046] The specific implementation process is as follows:
[0047] Step 1: Collect Cantonese opera segment data, and split the obtained Cantonese opera segment audio data into training set and test set;
[0048] Step 2: audio data preprocessing, including screening, tuning, slicing, framing, and windowing operations;
[0049] Step 3: Obtain the Mel spectrum of the audio data;
[0050] Step 4: Establish a Cantonese opera singing classification network model combining a multi-layer feature cascade convolutional neural network (CNN) and a long short-term memory unit (LSTM);
[0051] Step 5: Use the trained optimal classification network model to predict the Cantonese opera singing category and use the majority voting algorithm to predict the results.
[0052] In one embodiment, the step of preprocessing the Cantonese opera audio data sample set to obtain a Mel spectrum includes:
[0053] Slice the Cantonese opera audio data sample set into multiple audio files of equal length;
[0054] Performing a frame operation on the audio file, and performing a windowing operation on every two adjacent frame signals using a preset overlap rate;
[0055] Use fast Fourier transform to perform fast Fourier transform on each frame signal to obtain a spectrum diagram;
[0056] The spectrogram is processed by using a Mel filter bank to obtain a Mel spectrum.
[0057] In this embodiment, the data comes from Internet platforms such as China Opera Network, and the Cantonese opera singing style is used as the label of the data set. The data set uses five types of singing styles as the labels of the data set, including 165 Cantonese operas and a total of 500 singing style clips in WAV format. The length of each segment is 30s and the sampling rate is 44100Hz.
[0058] Since different Cantonese opera singing styles have many similarities, such as the instruments, rhythm and melody used for accompaniment, accurate classification requires screening out recognizable audio files. Considering the different sources of data sets for each style, slight changes are made to the original files. Without changing the singing style of the singing segment, the network's sensitivity to sound intensity is appropriately reduced, and the model's adaptability to audio of different loudness is improved. Therefore, it is very necessary to preprocess the original signal of the Cantonese opera singing segment. The analysis of audio signals includes screening, tuning, slicing, framing, etc.
[0059] Slicing: Use Python program to slice audio files of equal length (30s).
[0060] Framing: Speech signals are non-stationary signals (speech signals are generated by the vibration of vocal organs, but the movement state of vocal organs is unpredictable and affected by the environment). Studies have shown that within 10 to 30 ms, the spectral characteristics of speech signals can be considered basically unchanged (because the vibration audio of vocal tract muscles has a limit). Considering the internal continuity of Cantonese opera aria signals, in order to avoid the loss of certain information due to physical separation, overlapping framing is generally used. Figure 2 The audio signal framing diagram is shown, which shows the framing process of a 30-second Cantonese opera audio sample with a duration of 1.5 seconds and an overlap rate of 50% (there will be a 50% overlap between two adjacent frames), and the last part of the sample that is less than the duration is discarded.
[0061] The sampling FFT calculation obtains the spectrum information of the entire file. The FFT calculation with a sampling window length of 1024 obtains the spectrum information of the entire file, and each calculation slides forward 128 samples, so that the sampling windows overlap each other.
[0062] Acquisition of audio mel-spectrogram: Sound signals are originally one-dimensional time-domain signals, and it is difficult to see the frequency variation pattern intuitively. Using mel-spectrogram can more intuitively and specifically see the characteristics of related audio. Mel-spectrogram is a spectrogram under the mel scale, which is obtained by multiplying the spectrogram with several mel filters. The mapping relationship between the ordinary frequency scale and the mel frequency scale is shown in formula (1). Taking the Cantonese opera segment of Hong Xiannu as an example, the mapping relationship from Hz to mel in the Cantonese opera segment is as follows: Figure 3 As shown;
[0063]
[0064] The audio file is framed (long-time signals are framed into short-time signals) and windowed. The audio signal is mapped from the time domain to the frequency domain by performing Fourier transform (FFT) on each frame (the overlapping window part of the audio signal is performed). Finally, the results of each frame are stacked along another dimension to obtain a spectrum. Figure 4 shown.
[0065] Get Mel spectrum from audio signal: Convert the time domain signal to frequency domain by Fourier transform, and then use the Mel frequency scale filter bank to process the frequency domain signal to get Mel spectrum. Figure 5 The Mel spectrum of a Cantonese opera segment, the horizontal axis represents time, the vertical axis represents frequency, the color depth represents the energy, and the frequency scale of the Mel spectrum is exponentially distributed, as shown in Figure 1. Figure 6 shown.
[0066] In one embodiment, the classification network model constructed is as follows:
[0067] Implemented in Tensorflow2.2 using Keras API, the model network structure includes convolution layer, ReLU layer, Max-Pooling layer, Concat layer, LSTM layer, Flatten layer, Dense layer, Dropout layer, Batch Normalization layer, etc. The model consists of a two-level neural network, such as Figure 7 As shown. Taking the singing style, music and sound spectrum corresponding to the Cantonese opera segment as the model input, through the first-level network (Inception-CNN), composed of 2d-CRMD block and multi-scale feature extractor, the shallow and deep fusion feature expressions of each Cantonese opera segment can be obtained. Then through the second-level network (CNN-2LSTM), composed of CNN and stacked LSTM, the contextual association semantics between each segment and the importance of each segment are learned. At the same time, considering that the effective data set of Cantonese opera resources is relatively small, there is a certain possibility of overfitting relative to the expression ability and fitting ability of the network. The neural network model used in the present invention is connected with a dropout layer after the convolution layer, which can better reduce CNN overfitting.
[0068] First-level network: The CRMD block of the two-dimensional convolution converts the audio signal from the time domain to the frequency domain, and then uses the two-dimensional convolution to extract higher-level features from both the time and frequency directions. The multi-scale feature extractor uses a lightweight Inception structure to split the larger convolution through a symmetrical double-layer stacked convolution kernel structure, extracts features at different levels and fuses them to achieve multi-dimensional feature expression. Finally, the Concat layer connects the features of each dimension, maintaining a certain feature space information while enhancing the model's generalization ability for the feature range.
[0069] Secondary network: A feature cascade stacked network consisting of a double-layer LSTM and a convolutional layer. The double-layer LSTM stacked network extracts deep separable features layer by layer, from basic concept features to more abstract deep features. The stacked convolution kernels expand the learning range of the spectral features of the singing segment, which makes the network more capable of identifying potential patterns in the spectrum. Each layer outputs a feature vector sequence as the input of the subsequent layer, which is observed in blocks over time and expressed on different time scales, thereby enhancing the representation ability of the model. At the same time, it avoids the problem of "gradient disappearance" caused by the inability to iteratively update shallow weights due to the stacking of multiple LSTM layers.
[0070] In one embodiment, the Mel spectrum features are input into the classification network model for classification prediction, and the obtained prediction results are visualized using a confusion matrix, and the Cantonese opera singing category is predicted by a majority voting algorithm.
[0071] Using the MFCC of the original signal audio as the feature vector input, a Cantonese opera singing classification model combining a multi-layer feature cascade convolutional neural network (CNN) and a long short-term memory unit (LSTM) is established. The training set error and test set accuracy change curves of the model and the error change curves are shown in Figure 2. Figure 8-9 As shown in the figure, the optimal model obtained after training has an accuracy rate of more than 90% for both the training set and the test set, which effectively retains the characteristics of Cantonese opera singing while meeting the requirements of accurate classification.
[0072] The present invention plays an important role in the field of opera singing recognition and classification:
[0073] Learning audio through deep learning methods avoids the problems of manually selecting features, and to a certain extent avoids unnecessary waste of resources for those engaged in Cantonese opera. It can provide a more scientific, reasonable and efficient data management and classification method for the Cantonese opera database, and provide data support for the research on Cantonese opera performance forms, Cantonese opera singing schools and Cantonese opera music. It can also be applied to music software. Scientific classification brings better and more convenient listening experience to Cantonese opera lovers and listeners, and further promotes the inheritance of Cantonese opera culture.
[0074] The above is a detailed introduction to the Cantonese opera singing classification method based on the combination of CNN and LSTM provided by the present invention. This article uses specific examples to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for general technical personnel in this field, according to the idea of the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.
[0075] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A Cantonese opera singing classification method based on the combination of CNN and LSTM, characterized in that: The steps include: Steps to build a classification network model: Set the cascaded convolutional neural network Inception-CNN as the first-level network, set the CNN-two-layer LSTM neural network as the second-level network, and connect the output of the first-level network to the input of the second-level network; Steps to classify Cantonese opera singing using the classification network model: Through feature engineering, the Mel spectrum MFCC features are extracted from the Mel spectrum of the Cantonese opera audio signal of each segment; The Mel spectrum MFCC features are input into the primary network to fuse the shallow and deep features of the Cantonese opera audio signals of each segment, and the output signal is input into the secondary network; The secondary network learns the intrinsic features between the audio signals of each Cantonese opera segment, extracts the contextual semantics, and predicts the Cantonese opera singing category; The first-level network includes a two-dimensional convolutional CRMD block and a multi-scale feature extractor; The CRMD block of the two-dimensional convolution is composed of a Conv layer, a ReLU layer, a Maxpool layer and a Dropout layer; the CRMD block of the two-dimensional convolution converts the audio signal from the time domain to the frequency domain, and extracts multi-level features from both the time and frequency directions; the multi-scale feature extractor uses a lightweight Inception structure to split the multi-level features through a symmetrical double-layer stacked convolution kernel structure to extract features at different levels; The Concat layer connects the features of each level and inputs them into the secondary network; The secondary network includes a feature cascade stacking network formed by stacking a double-layer LSTM and a multi-convolutional layer CNN. The double-layer LSTM stacking network extracts deep separable features layer by layer, and each layer outputs a feature vector as the input of the subsequent layer.
2. According to the Cantonese opera singing classification method based on the combination of CNN and LSTM according to claim 1, it is characterized in that: The step of using the classification network model to classify Cantonese opera singing also includes the step of preprocessing the Cantonese opera audio data sample set to obtain the Mel spectrum, including: Slice the Cantonese opera audio data sample set into multiple audio files of equal length; Performing a frame operation on the audio file, and performing a windowing operation on every two adjacent frame signals using a preset overlap rate; Use fast Fourier transform to perform fast Fourier transform on each frame signal to obtain a spectrum diagram; The spectrogram is processed by using a Mel filter bank to obtain a Mel spectrum.
3. The Cantonese opera singing classification method based on the combination of CNN and LSTM according to claim 1 is characterized in that: The prediction results output by the classification network model are visually judged using a confusion matrix, and the Cantonese opera singing category is predicted using a majority voting algorithm.