Piano music automatic notation method for extracting music frequency structure
By introducing frequency convolution attention module and music frequency structure extraction network in the piano polyphonic music automatic notation system, using multi-head attention mechanism and masking matrix, the problem of insufficient frequency correlation modeling ability in the existing technology is solved, and the implementation of the realization of More accurate multi-pitch frequency correlation modeling and more efficient training improve the accuracy of notation results.
Patent Information
- Application Number
- CN202411007882.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-25
- Publication Date
- 2024-11-12
AI Technical Summary
The existing piano polyphonic music automatic notation method based on deep learning cannot fully utilize the frequency feature correlation in music signals, resulting in limited frequency correlation modeling ability and training efficiency, and the music frequency structure cannot be effectively extracted.
A multi-task learning framework automatic notation system for piano polyphonic music is proposed, using frequency convolution attention module, including frequency dynamic convolution network and music frequency structure extraction network, and introducing music frequency structure correlation through multi-head attention mechanism and masking matrix , combining pooling layer and linear layer for multitasking objective estimation.
This method can consider the frequency correlation between the fundamental frequency and the integer frequency multiplication and non-integer frequency multiplication at the same time, improves the modeling ability and training efficiency of pitch frequency characteristics, and achieves more accurate multi-pitch frequency correlation Modeling improves the accuracy of notation results.
Smart Images

Figure CN118942432A_ABST
Abstract
Description
Technical Field
[0001] The present invention is based on deep learning technology and invents a method for automatically notating piano music by extracting the frequency structure of music. First, a frequency dynamic convolution network is used to extract the time-frequency local features required for multiple note targets in the music time-frequency spectrum; then a music frequency structure extraction network is used to model the music frequency structure in each note target; finally, the frame-level target for estimating the note start time, the frame-level target for estimating the note stop time, the frame-level target for estimating the note multi-pitch, and the frame-level target for estimating the note intensity are estimated to obtain the notation result. The present invention belongs to the field of computer hearing, and specifically relates to technologies such as deep learning and automatic music notation. Background Art
[0002] Automatic Music Transcription (AMT) is one of the basic technologies in the field of music information retrieval, which aims to convert music audio signals into musical notation forms, such as simple notation, five-line notation, piano roll notation, and Musical Instrument Digital Interface (MIDI). Automatic music notation will improve the process of manual music notation, reduce the cost of manual music notation, and improve the efficiency of notation. In addition, with the help of automatic music notation technology, MIR tasks such as multi-pitch estimation, note starting point detection, beat detection and melody tracking can be achieved, which can provide support for application directions such as music theme classification, automatic music generation, music education, music-assisted creation, and musicology research.
[0003] Conducting targeted research using the piano as a typical instrument for polyphonic music notation is a key breakthrough in solving various difficulties in this technology. Recently, with the rapid development of deep learning technology, deep neural networks have shown great advantages in frame-level and note-level notation tasks. Among them, the more mature ones are automatic notation methods based on multi-task learning. This type of method uses the time-frequency spectrum of polyphonic piano music as the input of the deep neural network, and uses multiple deep neural network acoustic models to model multiple attributes of the notes to be estimated, and estimates multiple note estimation targets at the same time. Finally, the note integration post-processing method is used to obtain a complete notation result.
[0004] The pitch frequency characteristics of piano notes have obvious musical frequency structures. Making full use of these musical frequency structures will help achieve accurate pitch estimation. Among the existing methods for automatic notation of piano polyphonic music based on deep learning, there are methods that use dilated convolutional networks and Transformer networks to extract musical frequency structure information in piano music signals. However, due to the limitation of the number of constructed convolution kernels, the musical frequency structure that can be modeled by the dilated convolution method is relatively simple, which limits the frequency correlation modeling ability of the network. In addition, in the method based on the Transformer network, since no prior information about the correlation of musical frequency structure is introduced in the frequency attention calculation process, the network needs to capture and establish the connection between each frequency point from scratch, which limits the network frequency correlation modeling ability and training efficiency. In summary, these methods still cannot make full use of the frequency feature correlation in piano music signals. It is urgent to propose a method that can effectively extract the musical frequency structure in polyphonic music signals and then model the complete multi-pitch frequency correlation. Summary of the invention
[0005] The present invention proposes a multi-task learning piano polyphonic music automatic notation system for extracting music frequency structure in a multi-task learning framework, as shown in the attached Figure 1 As shown in the figure, Onset, Offset, Frame and Velocity represent the frame-level target for estimating the note onset time, the frame-level target for estimating the note off time, the frame-level target for estimating the note multi-pitch, and the frame-level target for estimating the note velocity. The system consists of the following deep neural networks:
[0006] (1) Frequency Convolution Attention Module
[0007] In each branch, the input feature map tensor will first pass through four frequency convolutional attention network modules in sequence. Each frequency convolutional attention network module contains a frequency dynamic convolutional network module and a music frequency structure extraction network module in sequence, as shown in the attached figure. Figure 2 shown.
[0008] (2) Output pooling layer and linear layer
[0009] Finally, before each multi-task target is output, the feature map tensor will undergo one-dimensional maximum pooling along the feature map frequency dimension F, and a linear layer that maps the feature map channel dimension C to the number of piano roll keys of the piano music automatic notation target to obtain the final multi-task piano polyphonic music automatic notation estimation result.
[0010] A piano music automatic notation method for extracting music frequency structure, characterized by comprising:
[0011] Obtain a constant Q transform time-frequency spectrogram of a piano music audio signal, and convert the time-frequency spectrogram into a piano roll score using a piano music automatic notation neural network that extracts music frequency structure, wherein the number of frequency points of the constant Q transform time-frequency spectrogram of the piano music audio signal is 352, and every 48 frequency points correspond to a music frequency span of one octave, and the number of keys of the piano roll score is 88;
[0012] A multi-branch automatic notation network is constructed for multiple piano music automatic notation targets, wherein the multiple piano music automatic notation targets include: a note attack time estimation frame level target, a note stop time estimation frame level target, a note multi-pitch estimation frame level target and a note velocity estimation frame level target.
[0013] The multi-branch automatic notation network has a piano music automatic notation neural network backbone that extracts the music frequency structure;
[0014] The piano music automatic notation neural network for extracting music frequency structure is composed of a plurality of frequency convolutional attention network modules connected in series, and a pooling layer and a linear layer are sequentially included between all the network modules connected in series and the output piano music automatic notation target;
[0015] The frequency convolutional attention network module is composed of a frequency dynamic convolutional network module and a music frequency structure extraction network module connected in series.
[0016] The frequency dynamic convolutional network module includes input feature map Through K = 8 parallel convolutional network layers Conv k Get K outputs Where C represents the number of channels of the feature map, L represents the number of frames of the time-frequency spectrogram contained in the feature map, F represents the number of frequency points of the time-frequency spectrogram contained in the feature map, and Conv k It is composed of a batch normalization (BatchNorm) layer, a two-dimensional convolutional network layer, a BatchNorm layer, and a sigmoid activation function layer. The convolution kernel size of the two-dimensional convolutional network layer is (7′7). 1,2,...,k The weights π of the samples and time information will be obtained through the compression and excitation network SE 1,2,...,k , and based on these weights, the K outputs Y 1,2,...,k Perform weighted summation and obtain the final frequency dynamic convolution output Among them, the weight extraction network based on the SE network sequentially includes an average pooling layer along the time dimension L, a one-dimensional convolutional network layer along the frequency dimension F, a BatchNorm layer, a sigmoid activation function layer, a linear layer, and a weight sum normalization layer based on the softmax function;
[0017] Music frequency structure extraction network module, including input feature map The frequency attention output feature map is obtained through h = 4 attention taps respectively In the frequency attention described above, for each attention tap, the attention value is calculated as follows:
[0018]
[0019] The f character in the lower right corner of all symbols indicates that the calculation of these attention values is based on the frequency dimension. are the query vector, key vector and value vector obtained by linear mapping of the input vector, d f For each attention tap, the tap dimension size corresponds to is the masking matrix; in the frequency attention, for all attention taps, the multi-head attention value is calculated as follows:
[0020] MultiHead(Q f ,K f ,V f )=Stack(head1;head2;...;head h )W f O (2)
[0021] in To integrate the information on each attention tap, the output linear transformation matrix is used to obtain the overall comprehensive frequency attention output result. Stack(·) represents the matrix concatenation operation along the channel dimension of the attention tap. For each head i ,i∈{1,2,...,h}, we have:
[0022] head i =RelativeAttention(Q fi ,K fi ,V fi ) (3)
[0023] Where i represents the attention tap number, And Q in formula (6) f ,K f ,V f Corresponding in form, the query vector, key vector, and value vector with a subscript i here represent independent query vectors, key vectors, and value vectors distributed on each attention tap.
[0024] Masking matrix M f include:
[0025] The masking matrix Mf The element values at the selected positions in the matrix are set to 0, and the element values at the remaining positions are set to negative infinity;
[0026] The masking matrix M f The element positions whose element values are set to 0 are selected in different ways on a total of h = 4 attention taps, with M fi ,i∈{1,2,...,h} represents different masking matrices on a total of h=4 attention taps, where i represents the sequence number of the attention tap;
[0027] The method for selecting the element position with the selected element value set to 0 on the first attention tap includes: f1 In the matrix, the elements on the diagonal are taken as the center, and the positions of these elements are selected. In the upper triangular matrix and the lower triangular matrix, the elements with a distance of 48, 76, 96, 111, 124, 135, 144, 152, and 159 from the diagonal elements are selected in each row, and the element values of the selected positions are set to 0, and the element values of the other unselected element positions are set to negative infinity;
[0028] The method for selecting the element position with the selected element value set to 0 on the second attention tap includes: f2 In the following triangular matrix, the elements in each row that are 48 away from the elements on the matrix diagonal are selected as the center, and the elements that are 48, 76, 96, 111, 124, 135, 144, 152, and 159 away from these elements are selected in each row, and the element values of the selected positions are set to 0, and the element values of the remaining unselected element positions are set to negative infinity;
[0029] The method for selecting the element position with the selected element value set to 0 on the third attention tap includes: f3 In the following triangular matrix, the elements in each row that are 76 away from the elements on the matrix diagonal are selected as the center, and the elements that are 48, 76, 96, 111, 124, 135, 144, 152, and 159 away from these elements are selected in each row, and the element values of the selected positions are set to 0, and the element values of the remaining unselected element positions are set to negative infinity;
[0030] The method for selecting the element position with the selected element value set to 0 on the fourth attention tap includes: f4In the following triangular matrix, the elements in each row that are 96 away from the elements on the matrix diagonal are selected as the center, and the elements that are 48, 76, 96, 111, 124, 135, 144, 152, and 159 away from these elements are selected in each row, and the element values of the selected positions are set to 0, and the element values of the remaining unselected element positions are set to negative infinity.
[0031] The said layer of pooling layer and the layer of linear layer are sequentially included between the series-connected network module and each multi-task target of the output piano music automatic notation; the said pooling layer performs one-dimensional maximum pooling along the frequency dimension F of the feature map, the pooling kernel size is 4, and the said linear layer maps the channel dimension C of the feature map to 1 or 128.
[0032] Compared with the prior art, the present invention has the following obvious advantages and beneficial effects:
[0033] (1) The frequency correlation between the fundamental frequency and the integer and non-integer multiples in the frequency characteristics of the piano note pitch is taken into account at the same time; (2) Prior information about the correlation of music frequency structure is introduced into the frequency attention mechanism; (3) The frequency attention masking matrix designed and used does not add any additional computing resource requirements, maintaining the computational efficiency of the original frequency attention. BRIEF DESCRIPTION OF THE DRAWINGS
[0034] Figure 1 A schematic diagram of an automatic piano music notation network that extracts the frequency structure of music
[0035] Figure 2 This is a schematic diagram of the frequency convolutional attention network structure and a schematic diagram of the frequency dynamic convolutional network structure, where Represents the element-by-element summation of the feature map tensor
[0036] Figure 3 Schematic diagram of the music frequency structure in the CQT spectrum with 12 frequency points per octave. (a) shows the fundamental frequency f0 and its integer multiples and integer fractional multiples. (b) to (d) show the situation when all the frequency components shown in (a) are converted to 1 / 2, 1 / 3 and 1 / 4 of their original frequency. DETAILED DESCRIPTION
[0037] According to the above description, the following is a specific implementation process, but the scope of protection of this patent is not limited to this implementation process.
[0038] Step 1: Data preprocessing settings
[0039] The training set, validation set and test set provided in the MAESTRO V3.0.0 data set are used to train and evaluate the performance of the deep neural network proposed in this embodiment. In this step, the librosa audio processing library is used to perform the following preprocessing operations on the original audio in the MAESTRO data set. First, the average value of the sampling points on all channels of the original stereo audio is converted to mono audio, and the sampling frequency of the mono audio is downsampled from the original 44.1-48kHz to 16kHz. Subsequently, the downsampled audio signal is framed. The frame length is 128 milliseconds during framing, and the frame shift is 20 milliseconds. After framing, the constant Q transform (Constant-Q Transform, CQT) spectrogram of the audio signal is calculated, and the window type used is the Hanning window. The center frequency of the lowest frequency point in the CQT spectrogram is 27.5Hz (i.e., the pitch frequency of the lowest pitch A0 scale in the piano), and each frame of the CQT spectrum contains 352 frequency points in total, and each octave contains 48 frequency points, that is, each frequency point occupies a quarter of a semitone. For the obtained complex domain CQT spectrogram, take its modulus and map it to the dB scale. During the conversion process, the time-frequency point with the highest energy in each piano audio is used as the 0dB reference. The information of the dynamic range of 80dB in the logarithmic CQT spectrum is retained based on this reference point. Finally, the logarithmic CQT spectrum with a dynamic range of [-80dB, 0dB] is translated to the interval of [-40dB, +40dB] to complete the data preprocessing step. In the actual training and testing process, the preprocessed logarithmic CQT spectrum needs to be segmented. During the training process, the deep neural network proposed in this embodiment performs notation estimation on the 400×352 CQT spectrum each time, which means that the sample contains 640 frames, and each frame contains 352 frequency points. During the test process, the deep neural network proposed in this embodiment performs notation prediction on the 3200×352 CQT spectrum each time, which means that the sample contains 640 frames, and each frame contains 352 frequency points. In addition, in order to ensure the continuity of the notation results of each sample during the testing phase, each sample is shifted by 1600 frames, and the middle 800 to 2399 frames of each input sample are taken as the corresponding output results each time. Finally, all the output results are spliced in sequence to obtain the final notated piano roll score result.
[0040] In addition, in order to conduct supervised training and evaluation tests on the deep neural network in this embodiment, the correct notation target corresponding to the piano audio data obtained by preprocessing is converted from the MIDI file contained in the MAESTRO data set. The preprocessing steps in this part are specifically as follows: First, each note event in the MIDI file is continuously annotated with its corresponding key start frame and key end frame in a frame-level piano roll score containing 88 key categories to obtain a frame-level annotation of multiple pitch estimates of the notes. The piano roll score is a two-dimensional matrix, one dimension of which represents the time frame sequence, and the other dimension represents the pitch size. In the annotated piano roll score, a value of 0 is used to indicate that no note appears, and a value of 1 is used to indicate that a note appears. Here, the frame-level target annotation indicates that a certain pitch appears at a certain moment. If the note event contains a sustain pedal event, the pause time of the note with a sustain pedal value not less than 64 is extended until the note of the same pitch is pressed again or the sustain pedal value is less than 64 to end the note, and the time frame corresponding to the pause time at this time is recorded as the correct note end frame. In the following description, the note end frame and the key end frame when there is no sustain pedal event are collectively referred to as the note end frame. Next, on the basis of the note multi-pitch estimation frame level annotation, all key attack frames are extracted as attack target annotations, and all note end frames are extracted as stop target annotations. Here, the attack target annotation and the stop target annotation respectively indicate the event that a certain pitch note starts to appear at a certain moment, and the event that a certain pitch note stops appearing at a certain moment. Finally, the note strike force multi-pitch estimation target value is annotated at each attack target annotation to obtain the note strike force multi-pitch estimation target annotation. Here, the note strike force multi-pitch estimation target annotation indicates the sound intensity of the note when a certain pitch note appears at a certain moment. Among the four correct notation targets, the note onset estimation frame-level target, the note stop estimation frame-level target, and the note multi-pitch estimation frame-level target are binary classification targets, including two categories, 0 and 1; the note velocity estimation frame-level target is a multi-classification target, including a total of 128 categories corresponding to the note velocity values in the range of [0,127].
[0041] Step 2: Model structure, model parameters and model training settings
[0042] Step 2.1: Overall model structure setting
[0043] This embodiment proposes a multi-task learning structure for multiple attributes of notes. The structure first has a backbone composed of frequency convolutional attention network modules in series, and each subtask is extracted by a subtask module corresponding to the note estimation target, which are the four note targets described in step 1, namely, the note start time estimation frame level target, the note stop time estimation frame level target, the note multi-pitch estimation frame level target and the note intensity estimation frame level target. The overall network structure is shown in the attached figure. Figure 1shown.
[0044] First, the backbone network takes the CQT spectrogram as input, and then uses the serial frequency convolutional attention network module to extract the time-frequency information in the input feature map. Subsequently, a pooling layer and a linear layer are used on multiple task branches to reduce the dimension of the feature map, and the corresponding note estimation target piano roll score output is obtained. When each branch outputs, the sigmoid function is used at the end of the Frame branch, Onset branch, and Offset branch to constrain the output value. The softmax function is used at the end of the Velocity branch to constrain the output value with the total number of categories of the note firing intensity target as the total number of items. In the entire network, the deep features of the backbone network will be converted into different estimated outputs by the multi-branch sub-network before the final output, so as to realize information sharing between multiple tasks and improve the overall estimation consistency of the network. Here, for There is a sigmoid function σ Sigmoid (x)∈(0,1) is:
[0045]
[0046] for There is a softmax function σ Softmax (x)∈(0,1) K for:
[0047]
[0048] Step 2.2: Frequency dynamic convolutional network module structure and parameter setting
[0049] In the frequency dynamic convolutional network module, the input feature map Through K = 8 parallel convolutional network layers Conv k Get K outputs Where C represents the number of channels of the feature map, L represents the number of frames of the time-frequency spectrogram contained in the feature map, F represents the number of frequency points of the time-frequency spectrogram contained in the feature map, and Conv k It consists of a batch normalization (BatchNorm) layer, a two-dimensional convolutional network layer, a BatchNorm layer, and a sigmoid activation function layer. The convolution kernel size of the two-dimensional convolutional network layer is (7′7). K output Y 1,2,...,k The weights π of the sample and time information are obtained through the Squeeze-and-Excitation Networks (SE) 1,2,...,k , and based on these weights, the K outputs Y 1,2,...,k Perform weighted summation and obtain the final frequency dynamic convolution output Among them, the weight extraction network based on the SE network sequentially includes an average pooling layer along the time dimension L, a one-dimensional convolutional network layer along the frequency dimension F, a BatchNorm layer, a sigmoid activation function layer, a linear layer and a weight sum normalization layer based on a softmax function. In this embodiment, the first frequency dynamic convolutional network module expands the number of input feature map channels from 1 to 16, and then the number of feature map channels remains unchanged at 16 in each subsequent frequency dynamic convolutional network module.
[0050] Step 2.3: Music frequency structure extraction network module model structure and parameter setting
[0051] In the music frequency structure extraction network module, input feature map The frequency attention output feature map is obtained through h = 4 attention taps respectively In the frequency attention described above, for each attention tap, the attention value is calculated as follows:
[0052]
[0053] The f character in the lower right corner of all symbols indicates that the calculation of these attention values is based on the frequency dimension. are the query vector, key vector and value vector obtained by linear mapping of the input vector, d f For each attention tap, the tap dimension size corresponds to is the masking matrix. In the frequency attention described above, for all attention taps, the multi-head attention value is calculated as follows:
[0054] MultiHead(Q f ,K f ,V f )=Stack(head1;head2;...;head h )W f O (7)
[0055] in To integrate the information on each attention tap, the output linear transformation matrix is used to obtain the overall comprehensive frequency attention output result. Stack(·) represents the matrix concatenation operation along the channel dimension of the attention tap. For each head i ,i∈{1,2,...,h}, we have:
[0056] head i =RelativeAttention(Q fi ,K fi ,V fi ) (5)
[0057] Where i represents the attention tap number, And Q in formula (6) f ,K f ,V f Corresponding in form, the query vector, key vector, and value vector with a subscript i here represent independent query vectors, key vectors, and value vectors distributed on each attention tap.
[0058] In the music frequency structure extraction network module, the masking matrix M is used f It will help introduce prior knowledge about the music frequency structure into the frequency attention mechanism. In the music frequency structure, for the fundamental frequency f0 of any pitch, there are frequency components whose frequencies are several integer multiples of the fundamental frequency, such as 2f0, 3f0, etc., and also include secondary frequency components whose frequencies are several integer fractional multiples of the fundamental frequency, such as In addition, the music frequency structure also contains frequency components that are higher than the fundamental frequency and are non-integer multiples of the fundamental frequency, such as Etc., as well as the sub-frequency components of the non-integer fractional multiples of the fundamental frequency corresponding to the multiples of the fundamental frequency, such as The above-mentioned musical frequency structure relationships are all included in the correlation of pitch-frequency features. By introducing a suitable masking matrix for the frequency attention mechanism, the musical frequency structure in piano music can be effectively modeled.
[0059] In the CQT spectrum containing 12 frequency points in each octave, the fundamental frequency and its integer multiples and integer fractional multiples that conform to the above music frequency structure are shown in the following figure. Figure 3 In (a), the vertical lines represent the 10th multiple of the fundamental frequency f0, and the lowest represents the 10th multiple of the fundamental frequency f0. The horizontal axis represents the distance between the double frequency and the sub-frequency and the fundamental frequency, and the distance is positive in the direction of increasing frequency. In the CQT spectrum containing 12 frequency points per octave, the frequency components that meet the fundamental frequency and its non-integer multiple fundamental frequency in the above music frequency structure are located in the adjacent Figure 3 (b) to (d) are indicated by vertical lines. Figure 3 (b) indicates that when the Figure 3 All the frequency components shown in (a) are converted to their original frequency magnitudes. In the case of , it includes the fundamental frequency, the frequency components higher than the fundamental frequency and the sub-frequency components lower than the fundamental frequency. Figure 3 (c) with Figure 3 (d) respectively means when the attached Figure 3 All the frequency components shown in (a) are converted to their original frequency magnitudes. and The situation at that time.
[0060] For the masking matrix used in the music frequency structure extraction network module, it follows the above-mentioned music frequency structure representation method. When calculating the frequency attention in the CQT spectrum containing 48 frequency points per octave, the masking matrix will select the element position that matches the music frequency structure. The following settings are made for the masking matrix used: Masking matrix M f The element positions whose element values are set to 0 are selected in different ways on a total of h = 4 attention taps, with M fi , i∈{1,2,...,h} represents different masking matrices on a total of h=4 attention taps, where i represents the number of the attention tap. The masking matrix M of F′F on the first attention tap f1 In the matrix, the elements on the diagonal are selected as the center. In the upper triangular matrix and the lower triangular matrix, the elements with a distance of 48, 76, 96, 111, 124, 135, 144, 152, and 159 from the diagonal elements are selected in each row, and the element values of the selected positions are set to 0, and the element values of the other unselected element positions are set to negative infinity; the masking matrix M of F′F on the second attention tap f2 In the following triangular matrix, the elements in each row that are 48 away from the elements on the diagonal of the matrix are selected as the center, and the elements with distances of 48, 76, 96, 111, 124, 135, 144, 152, and 159 from these elements are selected in each row, and the element values of the selected positions are set to 0, and the element values of the other unselected element positions are set to negative infinity; the masking matrix M of F′F on the third attention tap f3 In the following triangular matrix, the elements in each row that are 76 away from the elements on the diagonal of the matrix are selected as the center, and the elements with distances of 48, 76, 96, 111, 124, 135, 144, 152, and 159 from these elements are selected in each row, and the element values of the selected positions are set to 0, and the element values of the other unselected element positions are set to negative infinity; the masking matrix M of F′F on the fourth attention tap f4 In the following triangular matrix, the elements in each row that are 96 away from the elements on the matrix diagonal are selected as the center, and the elements that are 48, 76, 96, 111, 124, 135, 144, 152, and 159 away from these elements are selected in each row, and the element values of the selected positions are set to 0, and the element values of the remaining unselected element positions are set to negative infinity.
[0061] Step 2.4: Output pooling layer and linear layer model structure and parameter setting
[0062] Before each multi-task branch outputs the piano roll score, it sequentially includes a pooling layer and a linear layer. The pooling layer performs one-dimensional maximum pooling along the frequency dimension F of the feature map, and the pooling kernel size is 4. The linear layer maps the feature map channel dimension C to 1 or 128, corresponding to the note onset estimation frame level target, the note stop moment estimation frame level target, and the note multi-pitch estimation frame level target as binary classification targets, including two categories 0 and 1; the note velocity estimation frame level target is a multi-classification target, including a total of 128 categories corresponding to the note velocity values in the range of [0,127].
Claims
1. A method for automatically notating piano music by extracting the frequency structure of music, characterized in that: include: Obtain a constant Q transform time-frequency spectrogram of a piano music audio signal, and convert the time-frequency spectrogram into a piano roll score using a piano music automatic notation neural network that extracts music frequency structure, wherein the number of frequency points of the constant Q transform time-frequency spectrogram of the piano music audio signal is 352, and every 48 frequency points correspond to a music frequency span of one octave, and the number of keys of the piano roll score is 88; A multi-branch automatic notation network is constructed for multiple piano music automatic notation targets, wherein the multiple piano music automatic notation targets include: a note attack time estimation frame level target, a note stop time estimation frame level target, a note multi-pitch estimation frame level target and a note velocity estimation frame level target.
2. The method according to claim 1, characterized in that include: The multi-branch automatic notation network has a piano music automatic notation neural network backbone that extracts the music frequency structure; The piano music automatic notation neural network for extracting music frequency structure is composed of a plurality of frequency convolutional attention network modules connected in series, and a pooling layer and a linear layer are sequentially included between all the network modules connected in series and the output piano music automatic notation target; The frequency convolutional attention network module is composed of a frequency dynamic convolutional network module and a music frequency structure extraction network module connected in series.
3. According to the method of claim 2, the frequency dynamic convolutional network module includes an input feature map Through K = 8 parallel convolutional network layers Conv k Get K outputs Where C represents the number of channels of the feature map, L represents the number of frames of the time-frequency spectrogram contained in the feature map, F represents the number of frequency points of the time-frequency spectrogram contained in the feature map, and Conv k It is composed of a batch normalization (BatchNorm) layer, a two-dimensional convolutional network layer, a BatchNorm layer, and a sigmoid activation function layer. The convolution kernel size of the 2D convolutional network layer is (7′7); K outputs Y 1,2,...,k The weights π of the sample and time information will be obtained through the compression and excitation network SE 1,2,...,k , and based on these weights, the K outputs Y 1,2,...,k Perform weighted summation and obtain the final frequency dynamic convolution output Among them, the weight extraction network based on the SE network sequentially includes an average pooling layer along the time dimension L, a one-dimensional convolutional network layer along the frequency dimension F, a BatchNorm layer, a sigmoid activation function layer, a linear layer and a weight sum normalization layer based on the softmax function.
4. The method according to claim 2, characterized in that: Music frequency structure extraction network module, including input feature map The frequency attention output feature map is obtained through h = 4 attention taps respectively In the frequency attention described above, for each attention tap, the attention value is calculated as follows: The f character in the lower right corner of all symbols indicates that the calculation of these attention values is based on the frequency dimension. are the query vector, key vector and value vector obtained by linear mapping of the input vector, d f For each attention tap, the tap dimension size corresponds to is the masking matrix; in the frequency attention, for all attention taps, the multi-head attention value is calculated as follows: in To integrate the information on each attention tap, the output linear transformation matrix is used to obtain the overall comprehensive frequency attention output result. Stack(·) represents the matrix concatenation operation along the channel dimension of the attention tap. For each head i ,i∈{1,2,...,h}, we have: head i =RelativeAttention(Q fi ,K fi ,V fi ) (3) Where i represents the attention tap number, Q fi ,K fi , i∈{1,2,...,h}, and Q in formula (1) f ,K f ,V f Corresponding in form, the query vector, key vector, and value vector with a subscript i here represent independent query vectors, key vectors, and value vectors distributed on each attention tap.
5. The method according to claim 4, characterized in that Masking matrix M f include: The masking matrix M f The element values at the selected positions in the matrix are set to 0, and the element values at the remaining positions are set to negative infinity; The masking matrix M f The element positions whose element values are set to 0 are selected in different ways on a total of h = 4 attention taps, with M fi ,i∈{1,2,...,h} represents different masking matrices on a total of h=4 attention taps, where i represents the sequence number of the attention tap; The method for selecting the element position with the selected element value set to 0 on the first attention tap includes: f1 In the matrix, the elements on the diagonal are taken as the center, and the positions of these elements are selected. In the upper triangular matrix and the lower triangular matrix, the elements with a distance of 48, 76, 96, 111, 124, 135, 144, 152, and 159 from the diagonal elements are selected in each row, and the element values of the selected positions are set to 0, and the element values of the other unselected element positions are set to negative infinity; The method for selecting the element position with the selected element value set to 0 on the second attention tap includes: f2 In the following triangular matrix, the elements in each row that are 48 away from the elements on the matrix diagonal are selected as the center, and the elements that are 48, 76, 96, 111, 124, 135, 144, 152, and 159 away from these elements are selected in each row, and the element values of the selected positions are set to 0, and the element values of the remaining unselected element positions are set to negative infinity; The method for selecting the element position with the selected element value set to 0 on the third attention tap includes: f3 In the following triangular matrix, the elements in each row that are 76 away from the elements on the matrix diagonal are selected as the center, and the elements that are 48, 76, 96, 111, 124, 135, 144, 152, and 159 away from these elements are selected in each row, and the element values of the selected positions are set to 0, and the element values of the remaining unselected element positions are set to negative infinity; The method for selecting the element position with the selected element value set to 0 on the fourth attention tap includes: f4 In the following triangular matrix, the elements in each row that are 96 away from the elements on the matrix diagonal are selected as the center, and the elements that are 48, 76, 96, 111, 124, 135, 144, 152, and 159 away from these elements are selected in each row, and the element values of the selected positions are set to 0, and the element values of the remaining unselected element positions are set to negative infinity.
6. The method according to claim 2, characterized in that A pooling layer and a linear layer are sequentially included between the series network module and each multi-task target of the output piano music automatic notation; the pooling layer performs one-dimensional maximum pooling along the frequency dimension F of the feature map, the pooling kernel size is 4, and the linear layer maps the channel dimension C of the feature map to 1 or 128.
