Implementation method of Guqin dynamic spectrum
Through the guqin music information extraction, digitization and synchronization matching technology, the cumbersome problem of manual marking of guqin music scores is solved, efficient and accurate synchronization of guqin music scores is achieved, and learning and appreciation experience is improved.
Patent Information
- Application Number
- CN202411338071.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-25
- Publication Date
- 2024-12-31
AI Technical Summary
In the prior art, the digital processing of guqin music scores is not automated enough, and a large amount of manpower is required to manually mark the note position and time information, which is prone to errors, especially in complex music scores, resulting in the cumbersome and inaccurate sound spectrum synchronization process.
Using methods of guqin music information extraction, digitization of scores, normalized preprocessing of data and data synchronization matching, the time, pitch and tone characteristics of music notes are automatically extracted through convolutional neural network, LSTM model and algorithm recognition technology, and using JSON data and dynamic time regularization algorithm realize synchronous matching of music and piano scores.
It improves the accuracy and efficiency of guqin sound score synchronization, reduces manpower, ensures the stability and reliability of synchronous display, and provides a more intuitive learning and appreciation experience.
Smart Images
Figure CN119229855A_ABST
Abstract
Description
Technical Field
[0001] The invention relates to the field of computer information processing, and in particular to a method for realizing a guqin dynamic score. Background Art
[0002] Guqin is an ancient Chinese musical instrument. Jianzipu is a score that describes the fingering of Guqin. With the development of science and technology and the emergence of audio and video, the way of displaying and spreading Guqin art has changed. In the past, people often learned Guqin by reading music scores or audio and video. The emergence of Guqin sound and score synchronization as a new form of expression has greatly facilitated the learning of Guqin. It combines the originally isolated audio and video content with the music score and displays them synchronously. While playing the audio and video, the music score is displayed synchronously, which improves the learning efficiency.
[0003] The common method currently used is to manually synchronize the sound with the spectrum. First, the position of each note in the score is manually marked, and then the timeline information of each note in the audio is recorded, so that when the audio is played, the area of the corresponding note position is automatically highlighted.
[0004] The existing system has the following problems in actual use:
[0005] 1. The music score may be a picture or a simple line score picture. It is not digital content. It needs to be manually aligned to mark the position information of all notes. Then, the time information of the audio of each position is recorded to display this position area during playback. This requires a lot of manual labor and is prone to errors.
[0006] 2. The process of manually synchronizing the sound spectrum is extremely tedious. Not only do you need to carefully identify each note on the score and determine its exact position in the score, but you also need to accurately match these positions with the timeline in the audio. For complex guqin scores, there may be a large number of different notes and playing techniques, which makes the marking work more difficult. For example, some fast playing passages may have multiple notes appearing densely, and it takes a high degree of concentration and patience to accurately mark the position and time information of these notes.
[0007] Therefore, a method for implementing guqin dynamic score is needed to solve the above problems. Summary of the invention
[0008] Technical issues solved
[0009] In view of the shortcomings of the prior art, the present invention provides a method for implementing the dynamic score of guqin, which solves the problem that the music score in the prior art may be a picture or a simple line score picture, which is not digital content itself and needs to be manually aligned to mark the position information of all notes, and then the time information of the audio of each position is recorded to display this position area during playback, which requires a lot of manual labor and is prone to errors.
[0010] Technical Solution
[0011] To achieve the above objectives, the present invention is implemented through the following technical solutions: a method for implementing guqin dynamic score, including guqin music information extraction, guqin score digitization, data normalization preprocessing, data synchronization matching and guqin sound spectrum synchronization system, the guqin music information extraction specifically includes identifying music events, extracting note time, pitch and timbre characteristics, the guqin score digitization specifically includes recording the guqin score in a digital manner and encoding the reduced character score, the data normalization preprocessing specifically includes recording music information and guqin score note sequence data in a unified manner, and the data synchronization matching is to perform algorithmic matching on the sequence generated by the music and the sequence generated by the score.
[0012] This solution uses a series of precise steps to efficiently integrate the music information of the guqin with the score and achieve synchronous display. Guqin music information extraction can accurately capture the key features of the notes and provide basic data for subsequent synchronization operations. The digitization of guqin scores enables traditional scores to be processed and analyzed in digital form, improving the operability and storage convenience of the scores. Data normalization preprocessing ensures the consistency and standardization of music information and score data, facilitating subsequent algorithm processing. Data synchronization matching is the core link of the entire solution. Through a specific algorithm, the precise synchronization of music and score is achieved, greatly improving the experience of learning and appreciating the guqin.
[0013] Preferably, in the guqin music information extraction step, a spectrogram is generated through audio preprocessing, a convolutional neural network or other model is used to automatically identify music events and preliminarily extract relevant features, the spectrogram is split into frames with a length of 30ms and a frame shift of 20ms according to the sampled audio data, 50 image data are generated per second, the data is sent to an LSTM model, and the convolutional neural network and the fully connected model are used to train and identify event information in the audio.
[0014] In the process of extracting guqin music information, audio preprocessing is the first key step. By generating a spectrogram, the audio signal can be converted into an image form that is easier to analyze. Then, using advanced deep learning models such as convolutional neural networks to automatically identify the spectrogram, it is possible to efficiently extract music events and related features. Splitting the spectrogram according to specific frame lengths and frame shifts can improve the resolution and accuracy of the data. Feeding the data into the LSTM model and combining it with the convolutional neural network and the fully connected model for training and recognition can give full play to the advantages of different models and improve the accuracy and reliability of music event recognition.
[0015] Preferably, in the guqin music information extraction step, an algorithm is used to perform more advanced feature recognition of music events, including time feature extraction, using short-time Fourier transform or other time-frequency analysis methods to decompose the audio signal and extract time features in the audio signal, such as the start time and duration of a note; pitch feature extraction, using a pitch detection algorithm to identify the pitch within a specified time range in the audio signal and record the pitch value of each note; timbre feature extraction, using Mel-frequency cepstral coefficients or other timbre feature extraction methods to extract the timbre features of the audio signal, which are used to distinguish between scattered sounds, overtones, pressed notes and other timbres in the guqin.
[0016] The extraction of guqin music information requires not only preliminary event recognition, but also more advanced feature recognition. Time feature extraction can accurately determine the start time and duration of a note, providing key time information for synchronization operations. Pitch feature extraction can accurately identify the pitch in the audio signal through a specific pitch detection algorithm, providing an important basis for matching the score. Timbre feature extraction can distinguish the different timbres of the guqin, such as scattered sounds, overtones, and pressed notes, enriching the content of the music information and improving the accuracy and expressiveness of the synchronous display.
[0017] Preferably, in the step of digitizing the guqin score, json data is used to mark the sequence of the abbreviated score notes, and the json data contains the position, pitch and fingering of the abbreviated score notes, providing detailed score information for synchronous operation.
[0018] The digitization of guqin notation is an important step in achieving the synchronization of sound and spectrum. Using json data to mark the sequence of notes in the reduced notation can store the notation information in a structured manner, which is convenient for subsequent processing and analysis. json data can contain various attributes of the reduced notation, such as the position, pitch, fingering, etc. of the notes, providing detailed notation information for synchronization operations. By referring to the digitization method used in a specific patent, the accuracy and standardization of the digitization process can be ensured, and the quality and usability of the notation can be improved.
[0019] Preferably, in the data normalization preprocessing step, a unified sequence format is used to mark the two sequences. The information sequence extracted from the music contains the extracted note events, and each note contains time, pitch, timbre and other features; the feature extraction of the notes is performed on the digitized Guqin score to form a sequence, which contains other feature information of the non-time information in the first sequence. In the data normalization preprocessing process, the information sequence extracted from the music and the digitized Guqin score sequence are marked with a unified sequence format, which can ensure that the two sequences have the same format and structure in subsequent processing, which is convenient for algorithm matching. The information sequence extracted from the music contains time, pitch, timbre and other features, which can comprehensively describe the music events. The feature extraction of the notes is performed on the digitized Guqin score to form a sequence containing non-time information, which can complement the information sequence extracted from the music and improve the accuracy and integrity of the synchronization operation.
[0020] Preferably, in the data synchronization matching step, dynamic time warping or other matching algorithms are used to match the notes in sequence 2 to the positions of each time point in sequence 1, so as to achieve the matching process of the two sequences. Through the synchronization of the two sequences, the guqin audio and the music score are synchronously output. Data synchronization matching is the core step to achieve the synchronization of the guqin sound spectrum. Using matching algorithms such as dynamic time warping, the notes in the digitized guqin score sequence can be accurately matched with the time points in the information sequence extracted from the music, so as to achieve the synchronization of the two sequences. Through the synchronization of the two sequences, the guqin audio and the music score can be synchronously output, so that the learner can intuitively see the corresponding music score while enjoying the guqin music, thereby improving the learning efficiency and experience. The selection and optimization of the matching algorithm is the key to ensure the accuracy and stability of synchronization.
[0021] Preferably, the guqin sound spectrum synchronization system includes a guqin music information extraction module, a guqin score digitization module, a data normalization preprocessing module and a data synchronization matching module, each system is composed of multiple modules, and each module has a specific function. The guqin music information extraction module is responsible for extracting music events and related features, and providing basic data for synchronization operations. The guqin score digitization module converts traditional guqin scores into digital form for easy processing and analysis. The data normalization preprocessing module uniformly processes music information and score data to ensure data consistency and standardization. The data synchronization matching module realizes the synchronous output of music and guqin scores through a specific algorithm. Each module works together to realize the synchronous playback of guqin sound spectrum, providing strong support for guqin learning and appreciation.
[0022] Preferably, the guqin music information extraction module includes an audio preprocessing unit, a music event recognition unit and a feature extraction unit. The guqin music information extraction module is a key component of the system by generating a spectrogram and using a deep learning model and algorithm to identify music events and extract features. The audio preprocessing unit generates a spectrogram by preprocessing the audio signal, providing a basis for subsequent music event recognition and feature extraction. The music event recognition unit uses a deep learning model and algorithm to automatically identify music events, such as the start, end and pitch change of a note. The feature extraction unit further extracts relevant features of the music event, such as time, pitch, timbre, etc. These units cooperate with each other to efficiently extract key information of guqin music and provide accurate data support for synchronous operation.
[0023] Preferably, the guqin score digitization module uses a specific data format to encode the reduced word score to realize the digital recording of the guqin score. The guqin score digitization module uses a specific data format to encode the reduced word score, which can convert the traditional guqin score into a digital form for easy storage, transmission and processing. The selection of data format should take into account the characteristics and requirements of the guqin score to ensure the accuracy and readability of the encoding. The digitally recorded guqin score can be matched and synchronized with the music data extracted by the music information extraction module to realize the synchronous display of the guqin sound spectrum. At the same time, the digitized guqin score also provides new ways and methods for guqin teaching, research and inheritance.
[0024] Preferably, the data synchronization matching module accurately matches the sequence generated by the music and the sequence generated by the music score to ensure the synchronization of the music and the music score. The data synchronization matching module is the core module for realizing the synchronization of the guqin sound and spectrum. It uses a specific matching algorithm to accurately match the sequence generated by the music and the sequence generated by the music score to ensure the synchronization of the music and the music score. The matching algorithm should be efficient, accurate and stable, and can adapt to different music styles and music score types. Through dynamic synchronous display, learners can intuitively see the corresponding music score while enjoying the guqin music, improving the learning effect and experience. At the same time, the data synchronization matching module can also be personalized according to the needs of users, such as adjusting the synchronization speed, display mode, etc., to meet the needs of different users.
[0025] Beneficial Effects
[0026] The present invention provides a method for realizing guqin dynamic score, which has the following beneficial effects:
[0027] The present invention combines deep learning and algorithm recognition to analyze audio signals from different angles. Deep learning models such as convolutional neural networks (CNN) and long short-term memory networks (LSTM) can automatically learn the complex features of audio signals, capture the time changes of notes and the potential patterns of features such as pitch and timbre. The algorithm recognition part, such as short-time Fourier transform (STFT), pitch detection algorithm (YIN algorithm) and timbre feature extraction method (MFCC), can accurately extract the time, pitch and timbre features of audio signals based on specific mathematical principles. The two methods cross-validate each other, which can reduce the errors that may occur in a single method and greatly improve the accuracy of music information extraction. For example, when identifying the start time and duration of a note, the deep learning model may deviate due to the limitations of the training data, while algorithms such as STFT can more accurately determine these time features based on the principles of signal processing, and the mutual verification of the two can obtain more accurate results.
[0028] The single music information extraction method in the present invention may be affected by various factors, resulting in unstable results. The combination of deep learning and algorithm recognition can increase the stability and reliability of the model. The deep learning model has a strong generalization ability, but may be affected by data noise and overfitting. The algorithm recognition method is usually based on mature mathematical theories and algorithms and has a certain stability. Through cross-validation, when an abnormal result occurs in one method, it can be verified and adjusted by another method, thereby improving the stability and reliability of the entire model. For example, if the output result of the deep learning model on a specific audio clip is significantly different from the expectation, and the result of the algorithm recognition is relatively stable, the cause of the problem in the deep learning model can be analyzed, and adjustments and optimizations can be made. At the same time, in practical applications, this stability and reliability are crucial for tasks that require accurate music information, such as guqin sound spectrum synchronization, which can ensure the accuracy and stability of synchronization and improve user experience.
[0029] Different music information extraction methods in the present invention have their own advantages and applicable scenarios. The combination of deep learning and algorithm recognition provides a more flexible option for music information processing. In different situations, the appropriate method or combination of methods can be selected according to specific needs. For example, for complex music clips, the deep learning model may be better at capturing the overall music structure and characteristics; while for specific timbre differentiation tasks, algorithms such as MFCC may be more accurate. Through cross-validation and model fusion, the parameters of the model and the combination of methods can be flexibly adjusted according to different audio characteristics and task requirements to achieve the best music information extraction effect. This flexibility makes the method applicable to various types of guqin music and other audio processing tasks, providing a powerful tool for fields such as music research, teaching and creation. BRIEF DESCRIPTION OF THE DRAWINGS
[0030] Figure 1 A sample picture of an ancient guqin score of the guqin score of the present invention;
[0031] Figure 2 A schematic diagram of a simplified line notation version of the guqin notation of the present invention;
[0032] Figure 3 The ancient score of the present invention is being synchronously displayed, and the played notes are being highlighted;
[0033] Figure 4 The modern musical score of the present invention is being displayed synchronously, and the played notes are being displayed synchronously;
[0034] Figure 5 It is a schematic diagram of the operation flow of making the current sound spectrum synchronized piano score of the present invention;
[0035] Figure 6 This is an effect diagram of converting audio into a spectrum diagram of the present invention;
[0036] Figure 7 A schematic diagram of music information after music information extraction according to the present invention;
[0037] Figure 8 A schematic diagram of word reduction defined in the digital go programming language of the present invention;
[0038] Fig. 9 A schematic diagram of the corresponding relationship between the fret position information and the pitch of the guqin of the present invention;
[0039] Fig.10 It is a matching schematic diagram of the DTW algorithm of the present invention;
[0040] Fig.11 A schematic diagram of sequence synchronization matching of the present invention;
[0041] Fig.12 The present invention is a schematic diagram of the effect of synchronously playing the sound of the guqin and the music score. DETAILED DESCRIPTION
[0042] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention. Specific embodiment one:
[0044] like Figure 1-12As shown, a method for implementing guqin dynamic score, guqin music information extraction, guqin score digitization, data normalization preprocessing, data synchronization matching and guqin sound spectrum synchronization system, guqin music information extraction, specifically including identifying music events, extracting note time, pitch and timbre characteristics, guqin score digitization specifically including using digital method to record guqin score and encoding reduced character score, data normalization preprocessing specifically including using unified method to record music information and guqin score note sequence data, data synchronization matching is algorithm matching of music generated sequence and score generated sequence.
[0045] This solution uses a series of precise steps to efficiently integrate the music information of the guqin with the score and achieve synchronous display. Guqin music information extraction can accurately capture the key features of the notes and provide basic data for subsequent synchronization operations. The digitization of guqin scores enables traditional scores to be processed and analyzed in digital form, improving the operability and storage convenience of the scores. Data normalization preprocessing ensures the consistency and standardization of music information and score data, facilitating subsequent algorithm processing. Data synchronization matching is the core link of the entire solution. Through a specific algorithm, the precise synchronization of music and score is achieved, greatly improving the experience of learning and appreciating the guqin.
[0046] In the step of extracting guqin music information, a spectrogram is generated through audio preprocessing, and a convolutional neural network or other models is used to automatically identify music events and preliminarily extract relevant features. The spectrogram is split into frames with a length of 30ms and a frame shift of 20ms according to the sampled audio data. 50 image data are generated per second and the data is sent to the LSTM model, which is used in conjunction with the convolutional neural network and the fully connected model to train and identify event information in the audio.
[0047] In the process of extracting guqin music information, audio preprocessing is the first key step. By generating a spectrogram, the audio signal can be converted into an image form that is easier to analyze. Then, using advanced deep learning models such as convolutional neural networks to automatically identify the spectrogram, it is possible to efficiently extract music events and related features. Splitting the spectrogram according to specific frame lengths and frame shifts can improve the resolution and accuracy of the data. Feeding the data into the LSTM model and combining it with the convolutional neural network and the fully connected model for training and recognition can give full play to the advantages of different models and improve the accuracy and reliability of music event recognition.
[0048] In the guqin music information extraction step, algorithms are used to perform more advanced feature recognition of music events, including time feature extraction, using short-time Fourier transform or other time-frequency analysis methods to decompose the audio signal and extract the time features in the audio signal, such as the start time and duration of the note; pitch feature extraction, using a pitch detection algorithm to identify the pitch within a specified time range in the audio signal and record the pitch value of each note; timbre feature extraction, using Mel-frequency cepstral coefficients or other timbre feature extraction methods to extract the timbre features of the audio signal, which are used to distinguish the timbre of scattered sounds, overtones, pressed notes, etc. in the guqin.
[0049] The extraction of guqin music information requires not only preliminary event recognition, but also more advanced feature recognition. Time feature extraction can accurately determine the start time and duration of a note, providing key time information for synchronization operations. Pitch feature extraction can accurately identify the pitch in the audio signal through a specific pitch detection algorithm, providing an important basis for matching the score. Timbre feature extraction can distinguish the different timbres of the guqin, such as scattered sounds, overtones, and pressed notes, enriching the content of the music information and improving the accuracy and expressiveness of the synchronous display.
[0050] In the digitization step of guqin notation, json data is used to mark the sequence of notes in the abbreviated notation. The json data contains the position, pitch and fingering of the notes in the abbreviated notation, providing detailed notation information for synchronization operations.
[0051] The digitization of guqin notation is an important step in achieving the synchronization of sound and spectrum. Using json data to mark the sequence of notes in the reduced notation can store the notation information in a structured manner, which is convenient for subsequent processing and analysis. json data can contain various attributes of the reduced notation, such as the position, pitch, fingering, etc. of the notes, providing detailed notation information for synchronization operations. By referring to the digitization method used in a specific patent, the accuracy and standardization of the digitization process can be ensured, and the quality and usability of the notation can be improved.
[0052] In the data normalization preprocessing step, a unified sequence format will be used to mark the two sequences. The information sequence extracted from music contains the extracted note events, and each note contains time, pitch, timbre and other features; the feature extraction of the notes is performed on the digitized Guqin score to form a sequence, which contains other feature information of the non-time information in the first sequence. In the data normalization preprocessing process, the information sequence extracted from music and the digitized Guqin score sequence are marked with a unified sequence format, which can ensure that the two sequences have the same format and structure in subsequent processing, which is convenient for algorithm matching. The information sequence extracted from music contains time, pitch, timbre and other features, which can comprehensively describe the music events. The feature extraction of the notes on the digitized Guqin score forms a sequence containing non-time information, which can complement the information sequence extracted from music and improve the accuracy and completeness of the synchronization operation.
[0053] In the data synchronization matching step, dynamic time warping or other matching algorithms are used to match the notes in sequence 2 to the various time points in sequence 1, thereby achieving the matching process of the two sequences. Through the synchronization of the two sequences, the guqin audio and the music score are synchronously output. Data synchronization matching is the core step to achieve the synchronization of the guqin sound spectrum. Using matching algorithms such as dynamic time warping, the notes in the digitized guqin score sequence can be accurately matched with the time points in the information sequence extracted from the music to achieve the synchronization of the two sequences. Through the synchronization of the two sequences, the guqin audio and the music score can be synchronously output, so that learners can intuitively see the corresponding music score while enjoying the guqin music, improving learning efficiency and experience. The selection and optimization of the matching algorithm is the key to ensuring synchronization accuracy and stability.
[0054] The guqin sound-score synchronization system includes a guqin music information extraction module, a guqin score digitization module, a data normalization preprocessing module, and a data synchronization matching module. Each system consists of multiple modules, and each module has a specific function. The guqin music information extraction module is responsible for extracting music events and related features, providing basic data for synchronization operations. The guqin score digitization module converts traditional guqin scores into digital form for easy processing and analysis. The data normalization preprocessing module uniformly processes music information and score data to ensure data consistency and standardization. The data synchronization matching module realizes the synchronous output of music and score through a specific algorithm. The modules work together to realize the synchronous playback of guqin sound scores, providing strong support for guqin learning and appreciation.
[0055] The Guqin music information extraction module includes an audio preprocessing unit, a music event recognition unit, and a feature extraction unit. It is a key component of the system by generating a spectrogram and using deep learning models and algorithms to identify music events and extract features. The audio preprocessing unit generates a spectrogram by preprocessing the audio signal, providing a basis for subsequent music event recognition and feature extraction. The music event recognition unit uses deep learning models and algorithms to automatically identify music events, such as the start, end, and pitch change of notes. The feature extraction unit further extracts relevant features of music events, such as time, pitch, timbre, etc. These units work together to efficiently extract key information of Guqin music and provide accurate data support for synchronization operations.
[0056] The digitization module of Guqin score adopts a specific data format to encode the reduced score, realizing the digital recording of Guqin score. The digitization module of Guqin score adopts a specific data format to encode the reduced score, which can convert the traditional score into digital form for easy storage, transmission and processing. The selection of data format should take into account the characteristics and requirements of the score to ensure the accuracy and readability of the encoding. The digitally recorded score can be matched and synchronized with the music data extracted by the music information extraction module to realize the synchronous display of Guqin sound spectrum. At the same time, the digitized score also provides new ways and methods for Guqin teaching, research and inheritance.
[0057] The data synchronization matching module accurately matches the sequence generated by the music and the sequence generated by the piano score to ensure the synchronization of the music and the piano score. The data synchronization matching module is the core module for realizing the synchronization of the guqin sound and spectrum. It uses a specific matching algorithm to accurately match the sequence generated by the music and the sequence generated by the piano score to ensure the synchronization of the music and the piano score. The matching algorithm should be efficient, accurate and stable, and be able to adapt to different music styles and piano score types. Through dynamic synchronous display, learners can intuitively see the corresponding piano score while enjoying the guqin music, improving the learning effect and experience. At the same time, the data synchronization matching module can also be personalized according to user needs, such as adjusting the synchronization speed, display mode, etc., to meet the needs of different users. Specific embodiment 2:
[0059] like Figure 1-12 As shown, a piano sound spectrum synchronization implementation scheme of the present invention specifically includes the following steps:
[0060] S01. Guqin music information extraction: specifically includes identifying music events, extracting note time, pitch, timbre characteristics, etc.
[0061] S02. Digitization of Guqin notation: specifically including recording Guqin notation in digital form and encoding the reduced notation.
[0062] S03. Data normalization preprocessing: specifically including using a unified method to record music information and the note sequence data of the guqin score.
[0063] S04, data synchronization matching: algorithm matching of the sequence generated by the music and the sequence generated by the score. The guqin music information extraction is carried out in the following steps:
[0064] P01. Music event extraction: Use deep learning technology to identify events in music files. This process initially goes through audio preprocessing, including noise reduction and standardization, and generates a spectrogram to optimize subsequent data analysis. Next, a convolutional neural network (CNN) or other model is used to automatically identify music events (such as note start and end and pitch changes) and initially extract relevant features (such as pitch, rhythm, and timbre). The specific way to use the model is to convert the spectrogram, see Figure 6 , according to the sampled audio data, it is split into frames with a length of 30ms and a frame shift of 20ms, generating 50 picture data per second, and sending the data to the LSTM model, which is combined with the convolutional neural network and the fully connected model to train and identify event information in the audio.
[0065] The model here aims to extract and identify the event information of Guqin music from audio data. It consists of three main parts: CNN is used to extract local features of the spectrogram, LSTM is used to process time series information, and the fully connected layer is used for the final event classification and feature recognition.
[0066] Convolutional Neural Network (CNN) part:
[0067] 1. Input layer:
[0068] The input is the spectrum generated by preprocessing the audio data. The sampled audio data is split into 30ms frames, with a frame shift of 20ms, and 50 images are generated per second. Suppose the input data is
[0069]
[0070] Where N is the batch size, Cin is the number of input channels (usually the number of color channels or feature dimensions of a spectrogram), Hin and Win are the height and width of the input image, respectively.
[0071] 2. Convolutional layer:
[0072] Use multiple convolution kernels for feature extraction. The convolution operation can be expressed as:
[0073]
[0074] in and are the input and output of the lth layer respectively, * represents the convolution operation, W lis the convolution kernel weight, b l is the bias term, and f is the activation function, such as ReLU.
[0075] 3. Pooling layer:
[0076] A pooling layer is usually used after a convolutional layer to reduce the dimension of the feature map while retaining important feature information. For example, the maximum pooling can be expressed as:
[0077] Where maxpool represents the maximum pooling operation.
[0078] 4. Output layer:
[0079] The output of CNN is a series of feature vectors, set as These feature vectors will serve as input to the LSTM.
[0080] Long Short-Term Memory (LSTM) Network
[0081] 1. Input layer:
[0082] Receive the feature vector sequence output by CNN, and set the input to
[0083] Where N is the batch size, T is the number of time steps, and C in is the input feature dimension.
[0084] 2. LSTM unit:
[0085] The LSTM unit controls the flow of information through a gating mechanism, including the input gate i t 、Forget gate t , output gate o t and cell state c t , the update formula is as follows:
[0086] Forget gate: f t =σ(W f .[h t-1 ,x t ]+b f )
[0087] Input gate: f t =σ(W i .[h t-1 ,x t ]+b i )
[0088] Candidate cell state: c t1 =tanh(W c .[h t-1 ,x t ]+b c )
[0089] Cell Status:
[0090] c t =f t *c t-1 +i t *c t1
[0091] Output Gate:
[0092] o t =σ(W o .[h t-1 ,x t ]+b o )
[0093] Hidden state:
[0094] h t =o t *tanh(c t )
[0095] Where σ is the sigmoid function, W f , W i , W c , W o is the weight matrix, b f 、b i 、b c 、b o is the bias term, h t-1 is the hidden state at the previous moment, x t is the input at the current moment.
[0096] 3. Output layer:
[0097] The output of LSTM is another sequence of feature vectors. Let the input be
[0098] 4. Fully connected layer:
[0099] Use multiple fully connected layers for feature fusion and event classification,
[0100] Suppose the output of the lth fully connected layer is
[0101] Where W l is the weight matrix, b l is the bias term and f is the activation function.
[0102] 5. Output layer:
[0103] The last layer of the fully connected layer usually uses the Softmax activation function for event classification and outputs the event information in the audio. Let the output be: Y∈RN×K , where K is the number of event categories.
[0104] Model training:
[0105] 1. Loss function:
[0106] The cross entropy loss function is usually used to measure the difference between the model prediction and the true label. Let the true label be Y true ∈R N×K , then the loss function can be expressed as:
[0107]
[0108] 2. Optimization algorithm:
[0109] Use optimization algorithms (such as Adam, SGD, etc.) to minimize the loss function and update the weights and bias terms of the model. Through such a model architecture, we can effectively combine the local feature extraction capability of CNN, the time series processing capability of LSTM, and the classification capability of the fully connected layer to accurately train and identify the event information of guqin music from audio data.
[0110] P02. Note feature recognition: Use algorithms to perform more advanced feature recognition of music events in order to obtain sufficient feature information. Time feature extraction: Use short-time Fourier transform (STFT) or other time-frequency analysis methods to decompose audio signals and extract time features in audio signals, such as the start time and duration of notes. Pitch feature extraction: Use pitch detection algorithms (such as the YI N algorithm) to identify the pitch within a specified time range in the audio signal and record the pitch value of each note. Timbre feature extraction: Use Mel-frequency cepstral coefficients (MFCC) or other timbre feature extraction methods to extract the timbre features of audio signals, which are used to distinguish the timbre of scattered sounds, overtones, and pressed notes in the guqin. By using deep learning and algorithm recognition for identification and cross-validation, the accuracy of music information extraction can be greatly improved. Figure 7 The extracted guqin music information includes the identified basic attribute information.
[0111] Short Time Fourier Transform (STFT) is a commonly used time-frequency analysis method, which is widely used in audio signal processing. It can display audio signals in two dimensions, time and frequency, so as to extract the time features in the audio signal, such as the start time and duration of the note. Through STFT, the audio signal can be divided into several short time periods, and Fourier transform is performed on each time period to obtain the spectrum information within the time period. In this way, the frequency distribution of the audio signal at different time points can be observed. For example, for a piece of music, the time when different notes appear and their duration can be analyzed through STFT.
[0112] Pitch is an important feature of audio signals and is of great significance for music analysis and processing. Pitch detection algorithms (such as the YIN algorithm) can identify the pitch within a specified time range in an audio signal and record the pitch value of each note. The YIN algorithm is an autocorrelation-based pitch detection algorithm that determines the pitch by calculating the autocorrelation function of the audio signal. Specifically, the YIN algorithm first preprocesses the audio signal and then calculates the autocorrelation function of the signal. The autocorrelation function will have peaks at the pitch period, and by finding these peaks, the pitch of the audio signal can be determined.
[0113] Timbre is another important feature of audio signals, which can be used to distinguish different instruments or sounds. Mel Frequency Cepstrum Coefficients (MFCC) is a commonly used timbre feature extraction method.
[0114] MFCC is based on the characteristics of the human auditory system, converting the audio signal into the Mel frequency domain and then calculating the cepstral coefficients. The specific steps are as follows:
[0115] 1. Preprocess the audio signal, including pre-emphasis, framing and windowing.
[0116] 2. Perform fast Fourier transform (FFT) on each frame of signal to obtain the spectrum.
[0117] 3. Pass the spectrum through the Mel filter bank to obtain the Mel spectrum.
[0118] 4. Take the logarithm of the Mel spectrum and then perform discrete cosine transform (DCT) to obtain MFCC.
[0119] 5. By extracting MFCC, the timbre characteristics of the audio signal can be obtained. When distinguishing the timbre of scattered sound, overtones and pressed notes in the guqin, different MFCC feature values can be used to distinguish them. For example, the MFCC features of scattered sound may have a specific distribution in certain frequency ranges, while the MFCC features of overtones and pressed notes may be different in other frequency ranges.
[0120] Convolutional Neural Network (CNN) has shown powerful capabilities in audio processing. In audio feature extraction, CNN can automatically learn the features of audio signals without manually designing feature extractors.
[0121] For audio signals, CNN can process the time-frequency representation of audio through convolutional layers and pooling layers. The convolution kernel in the convolutional layer can extract local features in the audio signal, such as patterns within a specific frequency range. The pooling layer can downsample the features to reduce the amount of calculation and the number of parameters. The fully connected layer is usually located at the end of the convolutional neural network. It integrates the features extracted by the convolutional layer and the pooling layer and outputs the final classification result or feature representation.
[0122] In audio feature extraction, the fully connected layer can model the high-level features of audio. The researchers built a fully convolutional network for extracting melody from polyphonic music. The network uses the combined frequency and periodicity (CFP) representation as input features, and after being processed by the encoder and decoder, it finally outputs the melody extraction result through the fully connected layer. Experimental results show that the model achieves good performance in the melody extraction task with less computation and number of parameters.
[0123] Traditional audio feature extraction methods usually rely on manually designed features, such as MFCC, etc. In deep learning methods, the input data can be the time-frequency representation of the original audio signal, such as STFT, Mel spectrum, etc. This change enables the model to automatically learn the characteristics of the audio signal, improving the accuracy and robustness of feature extraction.
[0124] The network structure in deep learning methods is usually more complex than that in traditional methods. For example, the combination of convolutional layers, pooling layers, and fully connected layers in convolutional neural networks can effectively extract features at different levels of audio signals. In addition, some studies have introduced new network structures, such as residual connections and attention mechanisms, to further improve the performance of the model. The audio feature models constructed by deep learning methods usually have higher dimensions and richer information. These feature models can better represent the characteristics of audio signals, thereby improving the performance of audio processing tasks. For example, when distinguishing the timbre of Guqin, such as scattered sounds, overtones, and pressed sounds, traditional MFCC features may not be sufficient to accurately distinguish different timbres. The feature models constructed by deep learning methods can learn more complex timbre features, thereby improving the accuracy of timbre classification. In short, in audio signal processing, short-time Fourier transform or other time-frequency analysis methods, pitch detection algorithms, and timbre feature extraction methods can be used to effectively extract the time, pitch, and timbre features of audio signals. By constructing convolutional and fully connected models, the accuracy and robustness of audio feature extraction can be further improved. During the model building process, aspects such as input data, network structure and feature model will change. These changes enable the audio feature extraction technology to continue to develop and progress.
[0125] In the process of building an audio feature extraction model, changes in many aspects are involved.
[0126] Traditional audio feature extraction methods usually rely on manually designed features, such as MFCC, etc. In deep learning methods, the input data can be the time-frequency representation of the original audio signal, such as STFT, Mel spectrum, etc. This change enables the model to automatically learn the characteristics of the audio signal, improving the accuracy and robustness of feature extraction.
[0127] Using multi-resolution STFT as input, CNN automatically learns the features of different playing techniques. Compared with traditional hand-crafted features, this automatically learned feature can better capture the complex characteristics of audio signals.
[0128] The network structure in deep learning methods is usually more complex than that of traditional methods. For example, the combination of convolutional layers, pooling layers, and fully connected layers in convolutional neural networks can effectively extract features at different levels of audio signals. In addition, some studies have introduced new network structures, such as residual connections and attention mechanisms, to further improve the performance of the model. For example, when distinguishing the timbre of Guqin, such as scattered sounds, overtones, and pressed notes, traditional MFCC features may not be sufficient to accurately distinguish different timbres. The feature model constructed by deep learning methods can learn more complex timbre features, thereby improving the accuracy of timbre classification.
[0129] Among them, for step S02, json data can be used to mark the sequence of notes of the Jianzi score. The data contains multiple attributes of the Jianzi score. For details, please refer to the digitization method used in the patent "A method for digitizing and dynamically generating Guqin Jianzi characters".
[0130] Among them, for step S03, we will use a unified sequence format to mark the two sequences. The information sequence extracted from music contains the extracted note events, and each note contains time, pitch, timbre and other features; then the feature extraction of the notes is performed on the digitized Guqin score to form a sequence, which contains other feature information other than time information in the first sequence as much as possible to facilitate subsequent processing. The following example illustrates the specific data format of the two sequences and the attributes that need to be used. Figure 6 As shown, we use the sequence of event number, time, pitch, and timbre to record the information extracted from the audio. Similarly, we extract the number, pitch, and timbre information from the digitized piano score to generate two data sequences. This is to prepare for the next step of data synchronization. The example is as follows:
[0131] Audio Extraction Information:
[0132] [{1,0.25s,A,S},{2,0.5s,F,F},{3,1.25s,G,F},{4,2.5s,#B,S},{5,2.75s,D,A}]
[0133] The information sequence extracted from the piano score:
[0134] [{1,0},{2,A,S},{3,F,F},{4,G,F},{5,#B,S},{6,D,A}]
[0135] As shown above, there may be inconsistencies between the event numbers of audio recognition and the note sequence numbers of the piano score, which requires the use of a synchronization algorithm to map the sequence of the piano score to the sequence of the audio.
[0136] Among them, for step S04, we will use an algorithm to perform a synchronous matching algorithm on the two sequences of S03. Specifically, we can use dynamic time warping (DTW) or other matching algorithms to match the notes in sequence 2 to the positions of each time point in sequence 1, so as to achieve the matching process of the two sequences. By synchronizing the two sequences, the guqin audio and the music score can be output synchronously. For example, when playing to a fixed time, the corresponding notes in sequence 1 and the corresponding notes in the music score can be obtained. Calculate the similarity between feature sequences and find the best matching path. Dynamic synchronization, based on the matching results, dynamically synchronize the audio and video with the guqin score. While playing the audio, display the corresponding guqin score, and the notes and audio are displayed in real time. Specific embodiment three:
[0138] like Figure 1-12 As shown in the accompanying drawings, Figure 2 It shows the ancient zither score in picture format. Figure 2 shows a modern version of the tablature, including rhythm information, Figure 3 and Figure 4 It shows the effect of spectral synchronization in the past. Figure 5 The production process of spectral synchronization in the past is shown. First, the coordinate position information of each note in the picture needs to be manually circled, and then the time binding operation of each position is performed according to the time axis information of the audio to achieve the synchronization effect. This undoubtedly requires a lot of manpower to complete. Figure 2 contains rhythm information, Figure 3 Indicates that the notes being played are being highlighted. Figure 4 Indicates that the played notes are being displayed synchronously.
[0139] Through the present invention, accurate synchronous playback of guqin dynamic scores can be achieved, providing a more intuitive learning and playing experience. Specifically, the characteristic sequences in the audio and music scores are extracted, and then the two sequences are dynamically matched to achieve synchronization of the two sequences, thereby realizing the synchronization of the guqin's sound spectrum, which is completely automated and does not require manual marking.
[0140] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "comprising a reference structure" do not exclude the existence of other identical elements in the process, method, article or device including the elements.
[0141] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.
Claims
1. A method for realizing guqin dynamic score, including guqin music information extraction, guqin score digitization, data normalization preprocessing, data synchronization matching and guqin sound spectrum synchronization system, characterized in that: The guqin music information extraction specifically includes identifying music events, extracting note time, pitch and timbre features, the guqin score digitization specifically includes recording the guqin score in a digital manner and encoding the reduced score, the data normalization preprocessing specifically includes recording music information and the note sequence data of the guqin score in a unified manner, and the data synchronization matching is algorithmically matching the sequence generated by the music and the sequence generated by the score.
2. A method for realizing the guqin dynamic score according to claim 1, characterized in that: In the guqin music information extraction step, a spectrogram is generated through audio preprocessing, a convolutional neural network or other model is used to automatically identify music events and preliminarily extract relevant features, the spectrogram is split into frames with a length of 30ms and a frame shift of 20ms according to the sampled audio data, 50 image data are generated per second, and the data is sent to the LSTM model, which cooperates with the convolutional neural network and the fully connected model to train and identify event information in the audio.
3. A method for realizing the guqin dynamic score according to claim 1, characterized in that: In the guqin music information extraction step, an algorithm is used to perform more advanced feature recognition of music events, including time feature extraction, using short-time Fourier transform or other time-frequency analysis methods to decompose the audio signal and extract time features in the audio signal, such as the start time and duration of a note; pitch feature extraction, using a pitch detection algorithm to identify the pitch within a specified time range in the audio signal and record the pitch value of each note; timbre feature extraction, using Mel-frequency cepstral coefficients or other timbre feature extraction methods to extract the timbre features of the audio signal, which are used to distinguish the timbre of scattered sounds, overtones, pressed notes, etc. in the guqin.
4. A method for realizing the guqin dynamic score according to claim 1, characterized in that: In the digitization step of the guqin score, json data is used to mark the sequence of the abbreviated score notes. The json data contains the position, pitch and fingering of the abbreviated score notes, providing detailed score information for synchronization operations.
5. A method for realizing the guqin dynamic score according to claim 1, characterized in that: In the data normalization preprocessing step, a unified sequence format is used to mark two sequences. The information sequence extracted from music contains extracted note events, and each note contains time, pitch and timbre characteristics. Feature extraction of notes is performed on the digitized guqin score to form a sequence containing other feature information other than time information in the first sequence, which complements the information sequence extracted from music to improve the accuracy and completeness of the synchronization operation.
6. A method for realizing the guqin dynamic score according to claim 1, characterized in that: In the data synchronization and matching step, dynamic time warping or other matching algorithms are used to match the notes in sequence 2 to the various time points in sequence 1, thereby achieving a matching process for the two sequences. By synchronizing the two sequences, the guqin audio and music score are output synchronously.
7. A method for realizing the guqin dynamic score according to claim 1, characterized in that: The guqin sound spectrum synchronization system includes a guqin music information extraction module, a guqin score digitization module, a data normalization preprocessing module and a data synchronization matching module. Each module works together to achieve the synchronous playback of the guqin sound spectrum. The guqin music information extraction module is responsible for extracting music events and related features to provide basic data for synchronization operations. The Guqin score digitization module converts traditional Guqin scores into digital form for easy processing and analysis. The data normalization preprocessing module processes music information and Guqin score data in a unified manner to ensure data consistency and standardization. The data synchronization matching module achieves synchronous output of music and Guqin scores through a specific algorithm.
8. A method for realizing the guqin dynamic score according to claim 7, characterized in that: The guqin music information extraction module includes an audio preprocessing unit, a music event recognition unit and a feature extraction unit, which generates a spectrum diagram and uses a deep learning model and algorithm to recognize music events and extract features.
9. A method for realizing the guqin dynamic score according to claim 8, characterized in that: The guqin score digitization module adopts a specific data format to encode the reduced score to realize the digital recording of the guqin score.
10. A method for realizing the guqin dynamic score according to claim 9, characterized in that: The data synchronization matching module accurately matches the sequence generated by the music and the sequence generated by the piano score to ensure the synchronization between the music and the piano score.