A piano music notation method based on multi-channel information fusion of cyclic spectrum

Abstract

The present invention discloses a multi-channel information fusion piano music notation method based on cyclic spectrum, comprising the following steps: inputting the cyclic spectrum of a piano signal into a detection network with different targets to detect the starting point, end point, and duration of a piano note respectively; fusing the three channel features of the piano note starting point, end point, and duration to obtain an enhanced feature; performing dot product operations on the enhanced feature matrix with the three feature matrices of the note starting point, end point, and duration to obtain a fused feature matrix; then using a convolutional neural network to extract cross-channel information from the fused feature matrix; using a bidirectional gated recurrent unit to model the temporal correlation of the cross-channel information, and using a pitch sequence probability value output by a sigmoid activation function as a decision function to obtain a piano music note sequence, thereby achieving the purpose of piano music notation. The method of the present invention achieves high accuracy and robustness.

Description

Technical Field

[0001] The invention belongs to the field of music information retrieval and relates to a piano music notation method based on multi-channel information fusion of cyclic spectrum. Background Art

[0002] Piano notation, the process of extracting individual notes from piano music signals, is a fundamental and core problem in the field of music information retrieval. Piano music can contain multiple harmonically related notes simultaneously, each with a rich spectrum of harmonics, making the spectral structure of piano music highly complex. These components overlap, making it difficult to accurately extract the pitch information of the notes, making piano notation a significant challenge.

[0003] Existing piano notation methods include spectral decomposition-based methods and machine learning methods. Spectral decomposition-based methods include non-negative matrix decomposition, sparse decomposition, pseudo-two-dimensional spectral decomposition, and the like. The specific process is to use matrix decomposition to directly obtain the pitch matrix and activation matrix from the piano signal spectrum, and to achieve piano music notation by minimizing the minimum distance between the signal spectrum and the product of the pitch matrix and the activation matrix. However, spectral decomposition-based methods find it difficult to represent the properties of notes evolving over time, so their accuracy needs to be improved. The currently popular machine learning methods simultaneously consider factors such as the starting point and the duration of the note for modeling, but lack consideration of the harmonic complexity of the piano signal spectrum in terms of input features.

[0004] It is now necessary to explore piano notation methods suitable for processing piano music with multiple concurrent notes that satisfy harmonic relationships, so as to reduce the complexity of piano music representation and reduce the influence of harmonic components on note estimation. Summary of the Invention

[0005] In order to explore a piano notation method suitable for processing complex spectra, reduce the complexity of piano music representation, and reduce the influence of harmonic components on note estimation, the technical solution adopted by the present invention is: a piano music notation method based on multi-channel information fusion of cyclic spectrum, comprising the following steps:

[0006] The cyclic spectrum of the piano signal is input into a detection network with different objectives to detect the starting point, end point and duration of the piano note respectively;

[0007] The three channel features of piano note starting point, end point and note duration are fused to obtain enhanced features, so that the enhanced features pay attention to the information of note starting point, end point and note duration at the same time; the enhanced feature matrix is ​​subjected to dot product operation with the three feature matrices of note starting point, end point and note duration respectively to obtain a fused feature matrix;

[0008] Then use the convolutional neural network to extract the cross-channel information of the fusion feature matrix;

[0009] A bidirectional gated recurrent unit is used to model the temporal correlation of cross-channel information. The pitch sequence output by the bidirectional gated recurrent unit is converted into the probability value of the pitch sequence using the Sigmoid activation function. With the help of a decision function, the piano music note sequence is obtained from the pitch sequence probability value output by the Sigmoid activation function, thereby achieving the purpose of piano music notation.

[0010] Furthermore, the cyclic spectrum of the piano signal is obtained based on the following method:

[0011] The piano audio signal is divided into frames, and the cyclic spectrum density function of each frame signal is calculated to obtain the cyclic spectrum of the piano signal.

[0012] Furthermore: the cyclic spectral density function formula is as follows:

[0013]

[0014] Among them, S X (α, k) is a two-dimensional function of the cyclic frequency index α and the frequency index k. Let k = 0 in the above formula to obtain the cyclic spectrum of the frame signal. The cyclic spectrum is calculated frame by frame to generate the cyclic spectrogram of the entire audio.

[0015] Furthermore: the detection network includes a note start point detection subnetwork, a note end point detection subnetwork and a note duration detection subnetwork;

[0016] The note start point detection subnetwork, note end point detection subnetwork, and note duration detection subnetwork all use the same network structure of a four-layer convolutional neural network and a bidirectional gated recurrent unit cascade, and the parameters of each subnetwork are obtained by training with different objectives;

[0017] The note start point detection subnetwork, note end point detection subnetwork and note duration detection subnetwork all use cross entropy as the loss function, that is,

[0018]

[0019]

[0020]

[0021] T represents the number of frames, N = 88, which is the number of piano notes, I on (t,n)∈{0,1},I off (t,n)∈{0,1} and I fr (t,n)∈{0,1} represents the starting point, end point and note duration label value of the piano signal in the tth frame, P on (t,n)∈[0,1],P off(t,n)∈[0,1] and P fr (t,n)∈[0,1] represents the starting point, end point and note duration probability values ​​of the nth note in the tth frame output by each detection module;

[0022] Then the total loss function of the detection network is l tol The definition is as follows.

[0023] l tol =l on +l off +l fr (5)

[0024] Furthermore, the three note feature matrices of the piano note starting point, end point, and duration are first average pooled and max pooled in the channel dimension to generate average pooled and max pooled information matrices of the same dimension as the note feature matrix;

[0025] Then, a convolutional neural network is used to correlate the interactive features of the two information matrices to obtain an enhanced feature matrix with the same dimensions as the note start point, end point, and duration.

[0026] Then, the enhanced feature matrix is ​​dot-producted with the feature matrices of the note start point, end point, and duration to generate a fused feature matrix.

[0027] The specific formula is:

[0028] C avg =Avgpool(C1,C2,C3) (6)

[0029] C max =Maxpool(C1,C2,C3) (7)

[0030] C′ m =Sigmoid(Conv([C avg ,C max ]))⊙C m (8)

[0031] Where: m=1,2,3, They are the three channel feature matrices of note starting point, end point and note duration; represents the fusion feature matrix, Indicates the channel dimension of C m The average pooling information matrix obtained by average pooling is Indicates the channel dimension of C m Max pooling is performed to obtain the maximum pooling information matrix; Conv(·) represents the mapping function of the 3×3 convolution kernel of a single-layer convolutional neural network, Sigmoid is the activation function, and ⊙ represents the dot product operation.

[0032] Furthermore: the decision function expression is as follows:

[0033]

[0034] Where: p(i,j)∈[0,1] represents the probability value of the jth note in the i-th frame, Y final (i,j)∈{0,1} represents the final note output result, and Tr represents the decision threshold of the note probability.

[0035] A piano music notation device based on multi-channel information fusion of cyclic spectrum, comprising:

[0036] Detection module: used to input the cyclic spectrum of the piano signal into a detection network with different targets to detect the starting point, ending point and duration of the piano note respectively; the detection module includes a note starting point detection submodule, a note ending point detection submodule and a note duration detection submodule;

[0037] Fusion module: used to fuse the three channel features of piano note starting point, end point and note duration to obtain enhanced features, so that the enhanced features pay attention to the information of note starting point, end point and note duration at the same time; perform dot product operation on the enhanced feature matrix with the three feature matrices of note starting point, end point and note duration respectively to obtain the fused feature matrix;

[0038] Feature extraction module: used to extract cross-channel information of the fusion feature matrix using convolutional neural network;

[0039] Notation module: This module uses a bidirectional gated recurrent unit to model the temporal correlation of cross-channel information. It converts the pitch sequence output by the bidirectional gated recurrent unit into a probability value of the pitch sequence using a Sigmoid activation function. With the help of a decision function, the pitch sequence probability value output by the Sigmoid activation function is used to obtain a piano music note sequence, thereby achieving the purpose of piano music notation.

[0040] This invention provides a piano music notation method based on multi-channel information fusion using cyclic spectra. This method takes into account the distinct harmonic relationships between notes in piano music and the overlapping harmonic components of each note, resulting in a complex spectral structure and difficulty in extracting note pitch information. The invention uses cyclic spectra as input features for piano music signals and then performs note estimation based on the cyclic spectra to achieve piano music notation.

[0041] This method has the following advantages: cyclic spectrum representation of piano audio signals reduces interference from multiple harmonic components in the spectrum, enabling better extraction of note pitch information; and multi-channel information fusion of the outputs of different detection modules focuses on the characteristics of different channels, further optimizing the note sequence. Experimental results demonstrate that the proposed method achieves high accuracy and robustness. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0043] Figure 1 It is the piano notation method proposed in this application;

[0044] Figure 2 This is a time domain waveform of a piano signal;

[0045] Figure 3 The cyclic amplitude spectrum of the piano music signal;

[0046] Figure 4 is the result graph of note starting point feature detection;

[0047] Figure 5 is a note duration detection result diagram;

[0048] Figure 6 is the result graph of note endpoint feature detection;

[0049] Figure 7 is the predicted note probability map;

[0050] Figure 8 This is the piano notation result. DETAILED DESCRIPTION

[0051] It should be noted that, unless there is any conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0052] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. The following description of at least one exemplary embodiment is actually only illustrative and is in no way intended to limit the present invention and its application or use. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0053] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or combinations thereof.

[0054] Unless otherwise specifically stated, the relative arrangement of the parts and steps, numerical expressions and numerical values ​​described in these embodiments do not limit the scope of the present invention. At the same time, it should be clear that, for ease of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship. The techniques, methods and equipment known to ordinary technicians in the relevant fields may not be discussed in detail, but where appropriate, the techniques, methods and equipment should be considered as part of the authorization specification. In all examples shown and discussed here, any specific value should be interpreted as merely exemplary and not as a limitation. Therefore, other examples of the exemplary embodiments may have different values. It should be noted that similar numbers and letters represent similar items in the following figures, so once an item is defined in one figure, it does not need to be further discussed in subsequent figures.

[0055] In the description of the present invention, it should be understood that the directions or positional relationships indicated by directional words such as "front, back, up, down, left, right", "horizontal, vertical, vertical, horizontal" and "top, bottom" are usually based on the directions or positional relationships shown in the accompanying drawings. They are only for the convenience of describing the present invention and simplifying the description. Unless otherwise specified, these directional words do not indicate or imply that the device or element referred to must have a specific direction or be constructed and operated in a specific direction. Therefore, they cannot be understood as limiting the scope of protection of the present invention: the directional words "inside and outside" refer to the inside and outside relative to the outline of each component itself.

[0056] For ease of description, spatially relative terms such as "above", "above", "on the upper surface of", "above", etc. may be used herein to describe the spatial positional relationship of a device or feature to other devices or features as shown in the figures. It should be understood that spatially relative terms are intended to include different orientations of the device in use or operation in addition to the orientation described in the figures. For example, if the device in the drawings is inverted, the device described as "above other devices or structures" or "above other devices or structures" will be positioned as "below other devices or structures" or "below their position devices or structures". Thus, the exemplary term "above" can include both "above" and "below". The device can also be positioned in other different ways (rotated 90 degrees or in other orientations), and the spatially relative descriptions used here are interpreted accordingly.

[0057] In addition, it should be noted that the use of terms such as "first" and "second" to limit components is only for the convenience of distinguishing the corresponding components. Unless otherwise stated, the above terms have no special meaning and therefore cannot be understood as limiting the scope of protection of the present invention.

[0058] Figure 1 It is the piano notation method proposed in this application;

[0059] A piano music notation method based on multi-channel information fusion of cyclic spectrum includes the following steps:

[0060] S1: Input the cyclic spectrum of the piano signal into the detection network with different objectives to detect the starting point, end point and duration of the piano note respectively;

[0061] S2: Fusing the piano note starting point, end point, and note duration channel features to obtain enhanced features, so that the enhanced features simultaneously focus on the note starting point, end point, and note duration information; performing dot product operations on the enhanced feature matrix with the note starting point, end point, and note duration feature matrices to obtain a fused feature matrix;

[0062] S3: Use convolutional neural network to extract cross-channel information of fusion feature matrix;

[0063] S4: A bidirectional gated recurrent unit is used to model the temporal correlation of cross-channel information. The pitch sequence output by the bidirectional gated recurrent unit is converted into a probability value of the pitch sequence using a Sigmoid activation function. The pitch sequence probability value output by the Sigmoid activation function is used with the help of a decision function to obtain a piano music note sequence, thereby achieving the purpose of piano music notation.

[0064] Steps S1 / S2 / S3 / S4 are executed sequentially;

[0065] The piano music audio signal is downsampled to a sampling frequency of 16000 Hz. The signal is divided into frames, each frame is 0.128 ms long, and the interval between two adjacent frames is 0.032 ms. The cyclic spectrum of the piano music audio signal is discretized and the short-time Fourier transform X of each frame signal x(n) is calculated. N (k), where n represents the time domain serial number of the frame signal, N is the number of points of the short-time Fourier transform, and k represents the frequency point serial number corresponding to the short-time Fourier transform.

[0066] Furthermore, the cyclic spectrum of the piano signal is obtained based on the following method:

[0067] The piano audio signal is divided into frames, and the cyclic spectrum density function of each frame signal is calculated to obtain the cyclic spectrum of the piano signal.

[0068] Further: calculate the cyclic spectral density function of the frame signal, the cyclic spectral density function formula is as follows:

[0069]

[0070] Among them, S X (α, k) is a two-dimensional function of the cyclic frequency index α and the frequency index k. Let k = 0 in the above formula to obtain the cyclic spectrum of the frame signal. The cyclic spectrum is calculated frame by frame to generate the cyclic spectrogram of the entire audio.

[0071] In the piano music signal cyclic spectrogram obtained by (1), the five consecutive frames of cyclic spectrograms centered on the current frame are spliced ​​as the input features of the frame to generate the extended cyclic spectrogram of the entire audio, and input it into the note starting point, end point and note duration detection network respectively;

[0072] The dataset for this application uses the open source dataset MAPS, which is divided into a training set and a test set in a ratio of 8:1.

[0073] On the training set, the stochastic gradient descent algorithm is used to train the network weight parameters. The convolution kernel size is 3×3, the convolution step is set to 1, the optimizer is Adam, the learning rate is set to 0.001, the batch size is set to 256, and the optimal network parameters are obtained through continuous iteration.

[0074] The detection network includes a note start point detection subnetwork, a note end point detection subnetwork and a note duration detection subnetwork;

[0075] The note start point detection subnetwork, note end point detection subnetwork, and note duration detection subnetwork all use the same network structure of a four-layer convolutional neural network and a bidirectional gated recurrent unit cascade, and the parameters of each subnetwork are obtained by training with different objectives;

[0076] The cyclic spectrum is input into each sub-network, and the output of each sub-network is the probability value of the note starting point, end point and note duration. The cross entropy loss of formula (2) is the training target.

[0077] The training process is as follows: first randomly initialize the parameters of the convolutional neural network and the bidirectional gating unit, and then use the stochastic gradient descent method to iteratively optimize the parameters of each sub-network until the detection network converges.

[0078] The note start point detection subnetwork, note end point detection subnetwork and note duration detection subnetwork all use cross entropy as the loss function, that is:

[0079]

[0080]

[0081]

[0082] T represents the number of frames, N = 88, which is the number of piano notes, I on (t,n)∈{0,1},I off (t,n)∈{0,1} and I fr (t,n)∈{0,1} represents the starting point, end point and note duration label value of the piano signal in the tth frame, P on (t,n)∈[0,1],P off (t,n)∈[0,1] and P fr (t,n)∈[0,1] represents the starting point, end point and note duration probability values ​​of the nth note in the tth frame output by each detection module;

[0083] Use l on , l off and l fr They represent the loss functions of the note start point, end point and note duration detection modules calculated by formula (2), respectively. The total loss function of the detection network is l tol The definition is as follows.

[0084] l tol =l on +l off +l fr (5)

[0085] Furthermore, the process of fusing the features of the three channels of piano note starting point, end point and duration is as follows:

[0086] First, the three note feature matrices of the piano note starting point, end point, and note duration are average pooled and max pooled in the channel dimension respectively to generate average pooling and max pooling information matrices of the same dimension as the note feature matrix;

[0087] Then, a convolutional neural network is used to correlate the interactive features of the two information matrices to obtain an enhanced feature matrix with the same dimensions as the note start point, end point, and duration.

[0088] Then, the enhanced feature matrix is ​​dot-producted with the feature matrices of the note start point, end point, and duration to generate a fusion feature matrix.

[0089] The specific formula is:

[0090] C avg =Avgpool(C1,C2,C3) (6)

[0091] C max =Maxpool(C1,C2,C3) (7)

[0092] C′ m =Sigmoid(Conv([C avg ,C max ]))⊙C m (8)

[0093] Where: m=1,2,3, They are the three channel feature matrices of note starting point, end point and note duration; represents the fusion feature matrix, Indicates the channel dimension of C m The average pooling information matrix obtained by average pooling is Indicates the channel dimension of C m Max pooling is performed to obtain the maximum pooling information matrix; Conv(·) represents the mapping function of the 3×3 convolution kernel of a single-layer convolutional neural network, Sigmoid is the activation function, and ⊙ represents the dot product operation.

[0094] Furthermore, a two-layer convolutional neural network is used to extract the cross-channel information of the fused feature matrix. The convolution kernel size is set to 3×3, the step size is set to 1, the number of input channels is 3, and the number of output channels is set to 128.

[0095] Furthermore, a bidirectional gated recurrent unit is used to model the temporal correlation of cross-channel information and generate probability values ​​of note sequences of piano music.

[0096] The gated recurrent unit hidden layer is set to 256, the activation function is the Sigmoid function, and the BiGRU output is the probability value of the note sequence of piano music

[0097] Furthermore: the decision function expression is as follows:

[0098]

[0099] Where: p(i,j)∈[0,1] represents the probability value of the jth note in the i-th frame, Y final (i,j)∈{0,1} represents the final note output result, and Tr represents the decision threshold of the note probability.

[0100] A piano music notation device based on multi-channel information fusion of cyclic spectrum, comprising:

[0101] Detection module: used to input the cyclic spectrum of the piano signal into a detection network with different targets to detect the starting point, ending point and duration of the piano note respectively; the detection module includes a note starting point detection submodule, a note ending point detection submodule and a note duration detection submodule;

[0102] Fusion module: used to fuse the three channel features of piano note starting point, end point and note duration to obtain enhanced features, so that the enhanced features pay attention to the information of note starting point, end point and note duration at the same time; perform dot product operation on the enhanced feature matrix with the three feature matrices of note starting point, end point and note duration respectively to obtain the fused feature matrix;

[0103] Feature extraction module: used to extract cross-channel information of the fusion feature matrix using convolutional neural network;

[0104] Notation module: This module uses a bidirectional gated recurrent unit to model the temporal correlation of cross-channel information. It converts the pitch sequence output by the bidirectional gated recurrent unit into a probability value of the pitch sequence using a Sigmoid activation function. With the help of a decision function, the pitch sequence probability value output by the Sigmoid activation function is used to obtain a piano music note sequence, thereby achieving the purpose of piano music notation.

[0105] Figure 2 This is a time domain waveform of a piano signal, with the horizontal axis being time and the vertical axis being amplitude;

[0106] Figure 3 The cyclic amplitude spectrum of the piano music signal, with the horizontal axis being time and the vertical axis being the cyclic frequency;

[0107] Figure 4 This is the result of detecting the feature of the note starting point. The horizontal axis is time and the vertical axis is the MIDI value of the note.

[0108] Figure 5 This is the note duration detection result graph, with the horizontal axis being time and the vertical axis being the MIDI value of the note;

[0109] Figure 6 This is the result of the note endpoint feature detection. The horizontal axis is time and the vertical axis is the MIDI value of the note.

[0110] Figure 7 It is a predicted note probability graph, with the horizontal axis being time and the vertical axis being the MIDI value of the note;

[0111] Figure 8 This is the piano notation result graph, the horizontal axis is time, and the vertical axis is the MIDI value of the note.

[0112] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A piano music notation method based on multi-channel information fusion of cyclic spectrum, characterized by: The following steps are involved: The cyclic spectrum of the piano signal is input into a detection network with different objectives to detect the starting point, end point and duration of the piano note respectively; The three channel features of piano note starting point, end point and note duration are fused to obtain enhanced features, so that the enhanced features pay attention to the information of note starting point, end point and note duration at the same time; the enhanced feature matrix is ​​subjected to dot product operation with the three feature matrices of note starting point, end point and note duration respectively to obtain a fused feature matrix; Then use the convolutional neural network to extract the cross-channel information of the fusion feature matrix; A bidirectional gated recurrent unit is used to model the temporal correlation of cross-channel information. The pitch sequence output by the bidirectional gated recurrent unit is converted into the probability value of the pitch sequence using the Sigmoid activation function. With the help of a decision function, the piano music note sequence is obtained from the pitch sequence probability value output by the Sigmoid activation function, thereby achieving the purpose of piano music notation.

2. The piano music notation method based on multi-channel information fusion of cyclic spectrum according to claim 1, characterized in that: The cyclic spectrum of the piano signal is obtained based on the following method: The piano audio signal is divided into frames, and the cyclic spectrum density function of each frame signal is calculated to obtain the cyclic spectrum of the piano signal.

3. The piano music notation method based on multi-channel information fusion of cyclic spectrum according to claim 2, characterized in that: The cyclic spectral density function formula is as follows: （1） in, It is about the cycle frequency number , frequency number A two-dimensional function, let The cyclic spectrum of the frame signal is obtained, and the cyclic spectrum is calculated frame by frame to generate the cyclic spectrogram of the entire audio.

4. The piano music notation method based on multi-channel information fusion of cyclic spectrum according to claim 1, characterized in that: The detection network includes a note start point detection subnetwork, a note end point detection subnetwork and a note duration detection subnetwork; The note start point detection subnetwork, note end point detection subnetwork, and note duration detection subnetwork all use the same network structure of a four-layer convolutional neural network and a bidirectional gated recurrent unit cascade, and the parameters of each subnetwork are obtained by training with different objectives; The note start point detection subnetwork, note end point detection subnetwork and note duration detection subnetwork all use cross entropy as the loss function, that is, （2） （3） （4） Indicates the number of frames, , which is the number of notes on the piano, and Represents the piano signal in Frame No. The start point, end point, and duration label values ​​of each note, and Represents the output of each detection module. Frame No. The probability values ​​of the starting point, end point and duration of the note at each note; Then the total loss function of the detection network is The definition is as follows: （5）。 5. The piano music notation method based on multi-channel information fusion of cyclic spectrum according to claim 1, characterized in that: The process of fusing the features of the three channels of piano note starting point, end point and duration is as follows: First, the three note feature matrices of the piano note starting point, end point, and note duration are average pooled and max pooled in the channel dimension respectively to generate average pooling and max pooling information matrices of the same dimension as the note feature matrix; Then, a convolutional neural network is used to correlate the interactive features of the two information matrices to obtain an enhanced feature matrix with the same dimensions as the note start point, end point, and duration. Then, the enhanced feature matrix is ​​dot-producted with the feature matrices of the note start point, end point, and duration to generate a fused feature matrix. The specific formula is: （6） （7） （8） in: , They are the three channel feature matrices of note starting point, end point and note duration; represents the fusion feature matrix, Indicates the channel dimension The average pooling information matrix obtained by average pooling is Indicates the channel dimension Perform maximum pooling to obtain the maximum pooling information matrix; Represents a single-layer convolutional neural network The mapping function of the convolution kernel, is the activation function, Represents the dot product operation.

6. The piano music notation method based on multi-channel information fusion of cyclic spectrum according to claim 1, characterized in that: The decision function expression is as follows: （9） in: Indicates in Frame exists The probability value of a note, Indicates the final note output result, The decision threshold representing the note probability.

7. A piano music notation device based on multi-channel information fusion of cyclic spectrum, characterized by: include: Detection module: used to input the cyclic spectrum of the piano signal into a detection network with different targets to detect the starting point, ending point and duration of the piano note respectively; the detection module includes a note starting point detection submodule, a note ending point detection submodule and a note duration detection submodule; Fusion module: used to fuse the three channel features of piano note starting point, end point and note duration to obtain enhanced features, so that the enhanced features pay attention to the information of note starting point, end point and note duration at the same time; perform dot product operation on the enhanced feature matrix with the three feature matrices of note starting point, end point and note duration respectively to obtain the fused feature matrix; Feature extraction module: used to extract cross-channel information of the fusion feature matrix using convolutional neural network; Notation module: This module uses a bidirectional gated recurrent unit to model the temporal correlation of cross-channel information. It converts the pitch sequence output by the bidirectional gated recurrent unit into a probability value of the pitch sequence using a Sigmoid activation function. With the help of a decision function, the pitch sequence probability value output by the Sigmoid activation function is used to obtain a piano music note sequence, thereby achieving the purpose of piano music notation.

Images