Automatic music transcription method and system based on reference score

Abstract

This invention discloses an automatic music transcription method and system based on a reference musical score. It obtains the reference musical score corresponding to the audio to be transcribed and the prior bounding boxes of each note contained therein. Additionally, it extracts audio feature maps from the segmented images derived from the audio to be transcribed, and maps these audio feature maps to multiple predicted bounding boxes of notes. The difference between the predicted bounding box and its mapped prior bounding box for each segment is calculated as the uncertainty of the predicted bounding box. The K predicted bounding boxes with the smallest uncertainty are selected, and their corresponding audio features in the audio feature maps of the corresponding segment images are used to construct the initial query vector for each segment image. This initial query vector, which is closer to the actual note positions, guides the decoder's decoding process, focusing more attention on regions in the audio feature maps of the segment images that are more likely to contain notes, thus improving the accuracy of the audio transcription results.

Description

Technical Field

[0001] This invention belongs to the field of audio signal processing and music information retrieval technology, and more specifically, relates to an automatic music transcription method and system based on a reference musical score. Background Technology

[0002] In the automatic singing transcription task, converting singing audio into a sequence of notes is a fundamental challenge in the field of music information retrieval. The challenge mainly stems from the rich expressiveness of the human voice—singers naturally add vibrato, glissando, breath variations, and subtle pitch shifts. These characteristics make accurately extracting pitch and beat start from audio far more complex than processing synthesizer or pure instrument signals.

[0003] However, most existing deep learning-based transcription models are not designed for evaluation scenarios of known songs and fail to use standard musical scores as prior knowledge to guide the transcription process. This results in initial queries potentially falling into regions with ambiguous pitches or significant rhythmic deviations. For example, the model might incorrectly interpret a note that should be C4 as an surrounding D4 or B3, or misjudge ornaments as independent notes. The model still needs to blindly search across the entire pitch range and all possible rhythmic combinations. This not only significantly reduces transcription efficiency but, more importantly, is prone to making incorrect judgments due to local noise or overtone interference. Ultimately, this means that the system still cannot achieve the ideal robustness and real-time performance requirements in evaluation scenarios of known songs. Summary of the Invention

[0004] In view of the above-mentioned defects or improvement needs of the existing technology, the present invention provides an automatic music transcription method and system based on reference musical scores, thereby solving the technical problem of poor transcription accuracy and low transcription efficiency caused by erroneous judgments due to local noise or overtone interference.

[0005] To achieve the above objectives, according to a first aspect of the present invention, an automatic music transcription method based on a reference musical score is provided, comprising: The audio to be transcribed is converted into a spectral image, and the spectral image is segmented to obtain multiple image segments; Obtain the reference score corresponding to the audio to be transcribed, align the reference score with the audio to be transcribed in time, and extract the prior localization box of each note from the aligned reference score. Based on the transcription model, and combined with the prior bounding boxes of each note in the aligned reference score, the target bounding boxes of each note contained in each slice image are predicted. The target bounding boxes of each note contained in each segmented image are converted into the start time, end time and pitch of the corresponding note, and the start time, end time and pitch of each note contained in each segmented image are spliced ​​together to obtain the transcription result of the audio to be transcribed. Specifically, based on the transcription model and combined with the prior bounding boxes of each note in the aligned reference score, the target bounding boxes of each note contained in each slice image are predicted, including: The encoder based on the transcription model extracts audio features from each segment of the image to obtain the audio feature map of the corresponding segment image, and uses the prediction head to map the audio feature map of the corresponding segment image into prediction boxes of multiple notes; wherein, the prior localization box and the prediction box include the starting position coordinates of the note, the duration of the note and the frequency. The mapping relationship between the prior localization box of each note in the aligned reference score and the prediction box corresponding to each segment image is determined, and the difference between the prediction box corresponding to each segment image and its mapped prior localization box is calculated as the uncertainty of the corresponding prediction box. For each segmented image, based on the uncertainty of the prediction bounding box corresponding to the corresponding segmented image, the K prediction bounding boxes with the smallest uncertainty are selected to correspond to the audio features in the audio feature map of the corresponding segmented image, so as to construct the initial query vector of the corresponding segmented image. Based on the decoder of the transcription model, the initial query vector of the corresponding segmented image is used to perform attention transformation on the audio feature map of the corresponding segmented image. Based on the attention transformation result, the target bounding box of each note contained in the corresponding segmented image is predicted.

[0006] According to the above-described automatic music transcription method based on reference musical scores, the decoder of the transcription model includes multiple stacked Transformer decoding layers and an output layer. The Transformer decoding layer includes a self-attention module, a cross-attention module, and a feedforward neural network. Specifically, for any segment of an image, the self-attention module in the Transformer decoding layer performs a self-attention transformation on the initial query vector of that segment or the output of the feedforward neural network in the previous Transformer decoding layer to obtain the self-attention query vector of the current layer. The cross-attention module in the Transformer decoding layer performs a cross-attention transformation on the audio feature map of that segment based on the self-attention query vector of the current layer to obtain the cross-attention vector of the current layer. The feedforward neural network in the Transformer decoding layer performs a non-linear transformation on the cross-attention vector of the current layer and outputs it. The output layer maps the output of the feedforward neural network in the last Transformer decoding layer to obtain the target bounding boxes of each note contained in that segment of the image.

[0007] According to the above-mentioned automatic music transcription method based on reference musical scores, the loss function used by the transcription model during training includes standard bounding box regression loss and weighted classification loss. The standard bounding box regression loss is calculated based on the difference between the target bounding boxes of each note output by the transcription model for the sample slice image and the prior localization boxes of the corresponding notes in the sample reference musical score. The weighted classification loss is obtained by weighting and summing the classification losses of the corresponding target bounding boxes using the weights of the target bounding boxes output by the transcription model for the sample slice image. The weight of a target bounding box whose true type is a note is 1 + Uprior, where Uprior is the uncertainty of the target bounding box; the weight of a target bounding box whose true type is background is 1.

[0008] According to the above-mentioned automatic music transcription method based on reference musical scores, the step of converting the audio to be transcribed into a spectral image and segmenting the spectral image to obtain multiple segmented images specifically includes: The constant Q transform algorithm is used to perform time-frequency analysis on the audio to be transcribed to obtain the initial two-dimensional spectrum matrix of the audio to be transcribed. Logarithmically compress each element in the initial two-dimensional spectrum matrix, and then linearly map the logarithmically compressed elements to a preset integer range to obtain a post-processed two-dimensional spectrum matrix. The post-processed two-dimensional spectrum matrix of the single channel is copied to construct a three-channel RGB image, thus obtaining the spectrum image; Multiple silence segment points are located in the audio to be transcribed, and the spectral image is divided in the time dimension based on the multiple silence segment points to obtain multiple slice images.

[0009] According to the above-described automatic music transcription method based on a reference score, the spectral image is divided along the time dimension based on the multiple silence segmentation points to obtain multiple segmented images, specifically including: The multiple silent segment points are filtered to ensure that the filtered silent segment points satisfy the following condition: the difference between the time span between adjacent silent segment points and the frequency span between adjacent silent segment points is less than a preset value. The spectrum image is divided based on the selected silence segment points to obtain multiple segmented images.

[0010] Based on the above-mentioned automatic music transcription method based on reference musical scores, the target bounding box of any note is converted into the start time, end time, and pitch of the corresponding note using the following formula: ; ; ; Where Onset is the start time, Offset is the end time, and pitch is the pitch. , and These are the top-left x-coordinate, bottom-right x-coordinate, and bottom-right y-coordinate of the target bounding box for any given note. Where is the sampling frequency, hop_length is the number of sampling points between two adjacent data points in the initial two-dimensional spectrum matrix, fmin is the lowest frequency used by the constant Q transform algorithm, n_bins is the total number of frequency points in the initial two-dimensional spectrum matrix, and bins_per_octave is the preset time resolution.

[0011] According to the above-mentioned automatic music transcription method based on reference musical scores, the difference between the predicted bounding box corresponding to each segmented image and its mapped prior localization box is the Euclidean distance or L1 distance between the corresponding predicted bounding box and its mapped prior localization box.

[0012] According to a second aspect of the present invention, an automatic music transcription system based on a reference musical score is provided, comprising: An audio processing unit is used to convert the audio to be transcribed into a spectral image and to segment the spectral image to obtain multiple segmented images; The prior information acquisition unit is used to acquire the reference score corresponding to the audio to be transcribed, and after aligning the reference score with the audio to be transcribed in time, extract the prior location box of each note from the aligned reference score. The target bounding box prediction unit is used to predict the target bounding boxes of each note contained in each slice image based on the transcription model and in combination with the prior localization boxes of each note in the aligned reference score. The transcription result output unit is used to convert the target bounding boxes of each note contained in each segment image into the start time, end time and pitch of the corresponding note, and to perform time splicing on the start time, end time and pitch of each note contained in each segment image to obtain the transcription result of the audio to be transcribed. Specifically, based on the transcription model and combined with the prior bounding boxes of each note in the aligned reference score, the target bounding boxes of each note contained in each slice image are predicted, including: The encoder based on the transcription model extracts audio features from each segment of the image to obtain the audio feature map of the corresponding segment image, and uses the prediction head to map the audio feature map of the corresponding segment image into prediction boxes of multiple notes; wherein, the prior localization box and the prediction box include the starting position coordinates of the note, the duration of the note and the frequency. The mapping relationship between the prior localization box of each note in the aligned reference score and the prediction box corresponding to each segment image is determined, and the difference between the prediction box corresponding to each segment image and its mapped prior localization box is calculated as the uncertainty of the corresponding prediction box. For each segmented image, based on the uncertainty of the prediction bounding box corresponding to the corresponding segmented image, the K prediction bounding boxes with the smallest uncertainty are selected to correspond to the audio features in the audio feature map of the corresponding segmented image, so as to construct the initial query vector of the corresponding segmented image. Based on the decoder of the transcription model, the initial query vector of the corresponding segmented image is used to perform attention transformation on the audio feature map of the corresponding segmented image. Based on the attention transformation result, the target bounding box of each note contained in the corresponding segmented image is predicted.

[0013] According to a third aspect of the present invention, an electronic device is provided, comprising: a computer-readable storage medium and a processor; The computer-readable storage medium is used to store executable instructions; The processor is configured to read executable instructions stored in the computer-readable storage medium and execute the method as described in the first aspect.

[0014] According to a fourth aspect of the invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to perform the method as described in the first aspect.

[0015] According to a fifth aspect of the invention, a computer program product is provided, comprising a computer program or instructions that, when executed by a processor, implement the method as described in the first aspect.

[0016] In summary, compared with the prior art, the above-described technical solutions conceived by this invention can achieve the following beneficial effects: By acquiring the reference score corresponding to the audio to be transcribed and the prior bounding boxes of each note contained therein, the encoder based on the transcription model extracts the audio feature maps of the segmented images derived from the audio to be transcribed. The prediction head then maps the audio feature maps of the corresponding segmented images to multiple note bounding boxes. The difference between the predicted bounding box and its mapped prior bounding box for each segmented image is calculated as the uncertainty of the corresponding predicted bounding box. The K predicted bounding boxes with the smallest uncertainty are selected, and their corresponding audio features in the audio feature maps of the corresponding segmented images are used to construct the initial query vector for each segmented image. This initial query vector, which is closer to the actual note positions, guides the decoder's decoding process, focusing more attention on regions in the audio feature maps of the segmented images that are more likely to contain notes. This makes it easier for the decoder's attention adjustment to converge to the correct position, avoiding the attention mechanism getting stuck in local optima or diverging. This improves the accuracy of the transcription model for note detection and the accuracy of the transcription results obtained from the target bounding box transformation based on the transcription model's output. Attached Figure Description

[0017] Figure 1 This is a flowchart illustrating the automatic music transcription method based on a reference score provided in an embodiment of the present invention. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0019] This invention provides an automatic music transcription method based on a reference musical score, such as... Figure 1 As shown, it includes: Step 110: Convert the audio to be transcribed into a spectral image, and segment the spectral image to obtain multiple segmented images; Step 120: Obtain the reference score corresponding to the audio to be transcribed; after aligning the reference score with the audio to be transcribed in time, extract the prior localization box of each note from the aligned reference score. Step 130: Based on the transcription model, and combined with the prior bounding boxes of each note in the aligned reference score, predict the target bounding boxes of each note contained in each slice image. Step 140: Convert the target bounding boxes of each note contained in each segmented image into the start time, end time and pitch of the corresponding note, and perform time splicing on the start time, end time and pitch of each note contained in each segmented image to obtain the transcription result of the audio to be transcribed.

[0020] Specifically, based on the transcription model and combined with the prior bounding boxes of each note in the aligned reference score, the target bounding boxes of each note contained in each slice image are predicted, including: The encoder based on the transcription model extracts audio features from each segment of the image to obtain the audio feature map of the corresponding segment image, and uses the prediction head to map the audio feature map of the corresponding segment image into prediction boxes of multiple notes; wherein, the prior localization box and the prediction box include the starting position coordinates of the note, the duration of the note and the frequency. The mapping relationship between the prior localization box of each note in the aligned reference score and the prediction box corresponding to each segment image is determined, and the difference between the prediction box corresponding to each segment image and its mapped prior localization box is calculated as the uncertainty of the corresponding prediction box. For each segmented image, based on the uncertainty of the prediction bounding box corresponding to the corresponding segmented image, the K prediction bounding boxes with the smallest uncertainty are selected to correspond to the audio features in the audio feature map of the corresponding segmented image, so as to construct the initial query vector of the corresponding segmented image. Based on the decoder of the transcription model, the initial query vector of the corresponding segmented image is used to perform attention transformation on the audio feature map of the corresponding segmented image. Based on the attention transformation result, the target bounding box of each note contained in the corresponding segmented image is predicted.

[0021] Here, in step 110, the input audio to be transcribed undergoes time-frequency transformation and image encoding to obtain a spectral image, making it suitable for the vision-based transcription model. The audio to be transcribed is a performance audio. It should be noted that the transcription model can use existing object detection models. In this embodiment of the invention, an encoder-decoder-based object detection model will be used as the transcription model, transforming the transcription task into an object detection task for musical notes. The target bounding box can be represented as a four-element vector (x, y, w, h), where x and y are the horizontal and vertical coordinates of the starting position of the note, w is the duration of the note, and h is the frequency of the note.

[0022] In some embodiments, the original audio to be transcribed can be uniformly sampled, with the sampling rate set to 16kHz to ensure the accuracy of frequency domain analysis. Subsequently, the Constant Q Transform (CQT) algorithm is used to perform time-frequency analysis on the audio to be transcribed, obtaining the initial two-dimensional spectrum matrix. Unlike the Short-Time Fourier Transform, the CQT distributes the frequency in a logarithmic interval, which better reflects the semitone scale characteristics in musical perception. In some embodiments, the sampling frequency can be chosen to be consistent with the sampling frequency of the audio file; the number of sampling points between two adjacent frames in the CQT spectrum matrix can be set to 512, approximately 0.0116s; since the range of human voices generally does not exceed the range of a piano, the lowest frequency in the CQT transform can be set to the lowest piano note, 27.5Hz; the discrete frequencies within each octave can be set to 24 to achieve higher time resolution; to include as much spectral information as possible from the target notes, the total number of frequency points in the spectrum can be set to 178. Subsequently, each element in the initial two-dimensional spectrum matrix is ​​logarithmically compressed (i.e., each element is updated to its logarithm), and the logarithmically compressed elements are linearly mapped to a preset integer range (e.g., 0-255) to obtain a post-processed two-dimensional spectrum matrix with linear intensity mapping. Next, to adapt to the input format of mainstream object detection models (such as DETR), the single-channel post-processed two-dimensional spectrum matrix can be copied to construct a three-channel RGB image, where the values ​​of the three channels are identical, forming a color spectrum image. The size of this three-channel spectrum image can be represented as (number of time frames, number of frequency bands, 3).

[0023] Considering that the time dimension of the original spectrogram image is much larger than its frequency dimension, the spectrogram image can be segmented to obtain multiple approximately square-shaped slice images with time and frequency scales closer together. In some embodiments, multiple silence segment points in the audio to be transcribed can be located, and the spectrogram image can be divided along the time dimension based on these silence segment points to obtain multiple slice images. In other embodiments, these silence segment points can be filtered to ensure that the difference between the time span and the frequency span between adjacent silence segment points is less than a preset value. Then, the spectrogram image is divided along the time dimension based on the filtered silence segment points to obtain multiple slice images. Each slice image will serve as an independent input to the subsequent transcription model.

[0024] In step 120, prior localization information that can be used by the transcription model can be extracted using the reference score corresponding to the audio to be transcribed (i.e., the score referenced by the singer in the audio to be transcribed). The reference score can be stored in MusicXML format. Specifically, the reference score can be time-aligned with the audio to be transcribed, and the prior localization bounding box for each note can be extracted from the aligned reference score. Here, the prior localization bounding box includes the starting position coordinates, duration, and frequency of the corresponding note, which can also be represented as a four-element vector (x, y, w, h) consistent with the target bounding box.

[0025] Subsequently, the encoder based on the transcription model extracts audio features from each image segment, obtaining audio feature maps for the corresponding image segments. A prediction head (which can be a lightweight feedforward neural network) maps these audio feature maps to multiple note bounding boxes. Similar to the prior localization boxes, each predicted box includes the starting position coordinates, duration, and frequency of the note, and can be represented as a four-element vector (x, y, w, h). Next, the mapping relationship between the prior localization boxes of each note in the aligned reference score and the corresponding predicted boxes in each image segment is determined. For any given predicted box, the prior localization box that is closest to it in the time dimension can be selected to establish the mapping. Then, the difference between the predicted box corresponding to each image segment and its mapped prior localization box is calculated as the uncertainty Uprio of the corresponding predicted box. In some embodiments, the Euclidean distance or L1 distance between the predicted box and its mapped prior localization box can be calculated as the difference between the predicted box and its mapped prior localization box, thus obtaining the uncertainty of the predicted box. It is evident that the uncertainty of the prediction frame directly reflects the degree of deviation between the prediction result of the prediction head and the reference score.

[0026] For each image segment, based on the uncertainty of the predicted bounding boxes corresponding to that segment, the K predicted bounding boxes with the lowest uncertainty are selected, and their corresponding audio features in the audio feature map of the corresponding segment are used to construct the initial query vector for that segment. This guides the decoder of the transcription model to iterate from the audio features in the audio feature map that are closest to the actual note positions, thereby more accurately detecting the target bounding boxes of each note. Taking any image segment as an example (the processing flow for all image segments is the same and will not be repeated hereafter), the uncertainty of each predicted bounding box corresponding to that segment can be obtained and arranged in descending order of uncertainty, thus obtaining the K (e.g., 300) predicted bounding boxes with the lowest uncertainty. Then, based on the positions of these K predicted bounding boxes, the corresponding audio features are extracted from the audio feature map of that segment. Based on the extracted K audio features, the initial query vector for that segment is constructed by concatenating channels.

[0027] For any given image segment, the decoder of the transcription model uses the initial query vector of that image segment to perform an attention transformation on the audio feature map of that image segment, and then predicts the target bounding boxes of each note contained in that image segment based on the attention transformation result.

[0028] In some embodiments, the transcription model includes an encoder, a prediction head connected to the encoder, and a decoder. The encoder may be composed of multiple stacked Transformer encoding layers, and the decoder may include multiple stacked Transformer decoding layers and an output layer. Here, the Transformer decoding layer includes a self-attention module, a cross-attention module, and a feedforward neural network. For any segment of the image, the self-attention module in the Transformer decoding layer performs a self-attention transformation on the initial query vector of the segment or the output of the feedforward neural network in the previous Transformer decoding layer to obtain the self-attention query vector of the current layer. The cross-attention module in the Transformer decoding layer performs a cross-attention transformation on the audio feature map of the segment based on the self-attention query vector of the current layer (the query vector used in the cross-attention transformation is the self-attention query vector of the current layer, and the key vector and value vector are both the audio feature map of the segment), to obtain the cross-attention vector of the current layer. The feedforward neural network in the Transformer decoding layer performs a non-linear transformation on the cross-attention vector of the current layer and outputs it. The final output layer maps the output of the feedforward neural network in the last Transformer decoding layer to obtain the target bounding boxes of each note contained in the segment.

[0029] It is evident that using an initial query vector that is closer to the actual note position to guide the decoder's decoding process, thereby focusing more attention on the regions in the audio feature maps of the segmented image that are more likely to contain notes, makes it easier for the attention adjustment of the stacked Transformer decoding layers to converge to the correct position, avoiding the attention mechanism from getting stuck in local optima or diverging, thus improving the accuracy of the transcription model for note detection.

[0030] In other embodiments, the loss function used by the transcription model during training includes standard bounding box regression loss and weighted classification loss. The standard bounding box regression loss and weighted classification loss can be constructed by simple superposition or weighted summation, and this embodiment of the invention does not specifically limit this. Here, the standard bounding box regression loss is calculated based on the difference between the target bounding boxes of each note output by the transcription model for the sample slice image and the prior localization boxes of the corresponding notes in the sample reference score; the weighted classification loss is obtained by weighted summation of the classification losses of the corresponding target bounding boxes using the weights of the target bounding boxes output by the transcription model for the sample slice image. Specifically, the weight of the target bounding box whose true type is a note is 1 + Uprior, where Uprior is the uncertainty of the target bounding box, so that the transcription model pays more attention to target bounding boxes with higher uncertainty and greater deviation from the prior; the weight of the target bounding box whose true type is background is 1. It should be noted that the uncertainty of the target bounding box is calculated in the same way as the uncertainty of the predicted box, and is also determined based on the difference between the target bounding box and its matched prior localization box.

[0031] Specifically, during model training, the processing flow of the transcription model for sample segmented images (derived from sample audio, processed in the same way as step 110) is similar to the processing flow for segmented images of the audio to be transcribed given in the above embodiments. That is, the reference score corresponding to the sample audio is obtained and aligned, and then the prior localization box of each note is extracted from the aligned reference score; then, each sample segmented image is sequentially input into the transcription model. The encoder of the transcription model performs image semantic encoding on the input sample segmented images to obtain the audio feature map of the sample segmented image. Subsequently, the prediction head outputs multiple prediction boxes based on the audio feature map, and then calculates the uncertainty of each prediction box using its mapped prior localization box. Then, the audio features corresponding to the K prediction boxes with the lowest uncertainty are selected to construct an initial query vector. The decoder of the transcription model performs attention transformation on the audio feature map based on the initial query vector, and then outputs multiple target bounding boxes and their prediction probabilities (i.e., the probability that the model predicts that they are notes) based on the attention transformation result. Subsequently, based on the prior bounding boxes in the reference musical score fragment corresponding to the sample image segment, the true type of each target bounding box is determined to calculate the standard bounding box regression loss and weighted classification loss. It should be noted that during training, the model can be set to output a preset number of target bounding boxes (higher than the actual number of notes, e.g., 300). Then, based on the prior bounding boxes in the reference musical score fragment corresponding to the sample image segment, the Hungarian matching algorithm is used to match each target bounding box with the aforementioned prior bounding boxes to determine the true type of each target bounding box. Specifically, if any target bounding box matches any prior bounding box, its true type is a note; if any target bounding box cannot be matched, its true type is background.

[0032] In calculating the standard bounding box regression loss, target bounding boxes of true type "note" can be selected. The difference between each target bounding box and its matching prior localization box is then calculated to obtain the standard bounding box regression loss. In some embodiments, the standard bounding box regression loss can use GIoU loss to constrain the positional and dimensional consistency between the target bounding box and the prior localization box. In calculating the weighted classification loss, the classification loss of each target bounding box can be calculated, and then summed using the weights of each target bounding box. In some embodiments, the classification loss of the target bounding boxes can be calculated using Focal Loss.

[0033] The entire transcription model is trained end-to-end using backpropagation and gradient descent algorithms to minimize the aforementioned loss function. During training, the transcription model gradually learns to reduce the uncertainty between the predicted results and prior information, thereby generating more accurate transcription results.

[0034] During the model inference process, after the decoder of the transcription model outputs the target bounding boxes of each note contained in each segment image in sequence, the target bounding boxes of each note contained in each segment image are converted into the start time, end time and pitch of the corresponding note. Then, the start time, end time and pitch of each note contained in each segment image are spliced ​​together to obtain the transcription result of the audio to be transcribed.

[0035] In some embodiments, the target bounding box of any note can be converted into the start time, end time, and pitch of the corresponding note based on the following formula: ; ; ; Where Onset is the start time, Offset is the end time, and pitch is the pitch. , and These are the x-coordinates of the top-left corner, the x-coordinate of the bottom-right corner, and the y-coordinate of the bottom-right corner of the target bounding box for the note. is the sampling frequency, hop_length is the number of sampling points between two adjacent data points in the initial two-dimensional spectrum matrix, fmin is the lowest frequency used by the constant Q transform algorithm, n_bins is the total number of frequency points in the initial two-dimensional spectrum matrix, and bins_per_octave is the preset time resolution.

[0036] In summary, the automatic music transcription method provided by this invention obtains the reference score corresponding to the audio to be transcribed and the prior bounding boxes of each note contained therein. Furthermore, the encoder based on the transcription model extracts the audio feature maps of the segmented images derived from the audio to be transcribed. A prediction head maps the audio feature maps of the corresponding segmented images to multiple note prediction boxes, thereby calculating the difference between the prediction boxes corresponding to each segmented image and their mapped prior bounding boxes as the uncertainty of the corresponding prediction boxes. The K prediction boxes with the smallest uncertainty are selected, and their corresponding audio features in the audio feature maps of the corresponding segmented images are used to construct the initial query vectors for the corresponding segmented images. These initial query vectors, which are closer to the actual note positions, guide the decoder's decoding process, focusing more attention on the regions in the audio feature maps of the segmented images that are more likely to contain notes. This makes it easier for the decoder's attention adjustment to converge to the correct position, avoiding the attention mechanism getting stuck in local optima or diverging. This improves the accuracy of the transcription model for note detection and the accuracy of the transcription results obtained from the target bounding box transformation based on the transcription model's output.

[0037] The automatic music transcription system based on reference score provided by the present invention will be described below. The automatic music transcription system based on reference score described below can be referred to in correspondence with the automatic music transcription method based on reference score described above.

[0038] This invention provides an automatic music transcription system based on a reference score, comprising: An audio processing unit is used to convert the audio to be transcribed into a spectral image and to segment the spectral image to obtain multiple segmented images; The prior information acquisition unit is used to acquire the reference score corresponding to the audio to be transcribed, and after aligning the reference score with the audio to be transcribed in time, extract the prior location box of each note from the aligned reference score. The target bounding box prediction unit is used to predict the target bounding boxes of each note contained in each slice image based on the transcription model and in combination with the prior localization boxes of each note in the aligned reference score. The transcription result output unit is used to convert the target bounding boxes of each note contained in each segment image into the start time, end time and pitch of the corresponding note, and to perform time splicing on the start time, end time and pitch of each note contained in each segment image to obtain the transcription result of the audio to be transcribed.

[0039] Specifically, based on the transcription model and combined with the prior bounding boxes of each note in the aligned reference score, the target bounding boxes of each note contained in each slice image are predicted, including: The encoder based on the transcription model extracts audio features from each segment of the image to obtain the audio feature map of the corresponding segment image, and uses the prediction head to map the audio feature map of the corresponding segment image into prediction boxes of multiple notes; wherein, the prior localization box and the prediction box include the starting position coordinates of the note, the duration of the note and the frequency. The mapping relationship between the prior localization box of each note in the aligned reference score and the prediction box corresponding to each segment image is determined, and the difference between the prediction box corresponding to each segment image and its mapped prior localization box is calculated as the uncertainty of the corresponding prediction box. For each segmented image, based on the uncertainty of the prediction bounding box corresponding to the corresponding segmented image, the K prediction bounding boxes with the smallest uncertainty are selected to correspond to the audio features in the audio feature map of the corresponding segmented image, so as to construct the initial query vector of the corresponding segmented image. Based on the decoder of the transcription model, the initial query vector of the corresponding segmented image is used to perform attention transformation on the audio feature map of the corresponding segmented image. Based on the attention transformation result, the target bounding box of each note contained in the corresponding segmented image is predicted.

[0040] This invention provides an electronic device, including: a computer-readable storage medium and a processor; The computer-readable storage medium is used to store executable instructions; The processor is configured to read executable instructions stored in the computer-readable storage medium and execute the method as described in any of the above embodiments.

[0041] This invention provides a computer-readable storage medium storing computer instructions that cause a processor to perform the method described in any of the above embodiments.

[0042] This invention provides a computer program product, including a computer program or instructions, which, when executed by a processor, implement the method described in any of the above embodiments.

[0043] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method of automatic music transcription based on a reference score, characterized in that, include: The audio to be transcribed is converted into a spectral image, and the spectral image is segmented to obtain multiple image segments; Obtain the reference score corresponding to the audio to be transcribed, align the reference score with the audio to be transcribed in time, and extract the prior localization box of each note from the aligned reference score. Based on the transcription model, and combined with the prior bounding boxes of each note in the aligned reference score, the target bounding boxes of each note contained in each slice image are predicted. The target bounding boxes of each note contained in each segmented image are converted into the start time, end time and pitch of the corresponding note, and the start time, end time and pitch of each note contained in each segmented image are spliced ​​together to obtain the transcription result of the audio to be transcribed. Specifically, based on the transcription model and combined with the prior bounding boxes of each note in the aligned reference score, the target bounding boxes of each note contained in each slice image are predicted, including: The encoder based on the transcription model extracts audio features from each segment of the image to obtain the audio feature map of the corresponding segment image, and uses the prediction head to map the audio feature map of the corresponding segment image into prediction boxes of multiple notes; wherein, the prior localization box and the prediction box include the starting position coordinates of the note, the duration of the note and the frequency. The mapping relationship between the prior localization box of each note in the aligned reference score and the prediction box corresponding to each segment image is determined, and the difference between the prediction box corresponding to each segment image and its mapped prior localization box is calculated as the uncertainty of the corresponding prediction box. For each segmented image, based on the uncertainty of the prediction bounding box corresponding to the corresponding segmented image, the K prediction bounding boxes with the smallest uncertainty are selected to correspond to the audio features in the audio feature map of the corresponding segmented image, so as to construct the initial query vector of the corresponding segmented image. Based on the decoder of the transcription model, the initial query vector of the corresponding segmented image is used to perform attention transformation on the audio feature map of the corresponding segmented image. Based on the attention transformation result, the target bounding box of each note contained in the corresponding segmented image is predicted.

2. The automatic music transcription method based on a reference musical score according to claim 1, wherein, The decoder of the transcription model includes multiple stacked Transformer decoding layers and an output layer. Each Transformer decoding layer includes a self-attention module, a cross-attention module, and a feedforward neural network. For any image segment, the self-attention module in the Transformer decoding layer performs a self-attention transformation on the initial query vector of that image segment or the output of the feedforward neural network in the previous Transformer decoding layer to obtain the self-attention query vector of the current layer. The cross-attention module in the Transformer decoding layer performs a cross-attention transformation on the audio feature map of that image segment based on the self-attention query vector of the current layer to obtain the cross-attention vector of the current layer. The feedforward neural network in the Transformer decoding layer performs a non-linear transformation on the cross-attention vector of the current layer and outputs it. The output layer maps the output of the feedforward neural network in the last Transformer decoding layer to obtain the target bounding boxes of each note contained in that image segment.

3. The automatic music transcription method based on a reference musical score according to claim 1, wherein, The loss function used by the transcription model during training includes standard bounding box regression loss and weighted classification loss. The standard bounding box regression loss is calculated based on the difference between the target bounding boxes of each note output by the transcription model for the sample slice image and the prior localization boxes of the corresponding notes in the sample reference musical score. The weighted classification loss is obtained by weighting and summing the classification losses of the corresponding target bounding boxes using the weights of the target bounding boxes output by the transcription model for the sample slice image. Specifically, the weight of a target bounding box whose true type is a note is 1 + Uprior, where Uprior is the uncertainty of the target bounding box; the weight of a target bounding box whose true type is background is 1.

4. The automatic music transcription method based on a reference musical score according to any one of claims 1 to 3, characterized in that, The process of converting the audio to be transcribed into a spectral image and segmenting the spectral image to obtain multiple image segments specifically includes: The constant Q transform algorithm is used to perform time-frequency analysis on the audio to be transcribed to obtain the initial two-dimensional spectrum matrix of the audio to be transcribed. Logarithmically compress each element in the initial two-dimensional spectrum matrix, and then linearly map the logarithmically compressed elements to a preset integer range to obtain a post-processed two-dimensional spectrum matrix. The post-processed two-dimensional spectrum matrix of the single channel is copied to construct a three-channel RGB image, thus obtaining the spectrum image; Multiple silence segment points are located in the audio to be transcribed, and the spectral image is divided in the time dimension based on the multiple silence segment points to obtain multiple slice images.

5. The automatic music transcription method based on a reference musical score according to claim 4, wherein, The spectral image is divided along the time dimension based on the multiple silence segmentation points to obtain multiple segmented images, specifically including: The multiple silent segment points are filtered to ensure that the filtered silent segment points satisfy the following condition: the difference between the time span between adjacent silent segment points and the frequency span between adjacent silent segment points is less than a preset value. The spectrum image is divided based on the selected silence segment points to obtain multiple segmented images.

6. The automatic music transcription method based on a reference musical score according to claim 4, wherein, The target bounding box of any note is converted into the start time, end time, and pitch of the corresponding note based on the following formula: ； ； ； Where Onset is the start time, Offset is the end time, and pitch is the pitch. , and These are the top-left x-coordinate, bottom-right x-coordinate, and bottom-right y-coordinate of the target bounding box for any given note. Where is the sampling frequency, hop_length is the number of sampling points between two adjacent data points in the initial two-dimensional spectrum matrix, fmin is the lowest frequency used by the constant Q transform algorithm, n_bins is the total number of frequency points in the initial two-dimensional spectrum matrix, and bins_per_octave is the preset time resolution.

7. The automatic music transcription method based on a reference score as described in any one of claims 1 to 3, characterized in that, The difference between the predicted bounding box corresponding to each image segment and its mapped prior bounding box is the Euclidean distance or L1 distance between the corresponding predicted bounding box and its mapped prior bounding box.

8. An automatic music transcription system based on a reference score, characterized in that, include: An audio processing unit is used to convert the audio to be transcribed into a spectral image and to segment the spectral image to obtain multiple segmented images; The prior information acquisition unit is used to acquire the reference score corresponding to the audio to be transcribed, and after aligning the reference score with the audio to be transcribed in time, extract the prior location box of each note from the aligned reference score. The target bounding box prediction unit is used to predict the target bounding boxes of each note contained in each slice image based on the transcription model and in combination with the prior localization boxes of each note in the aligned reference score. The transcription result output unit is used to convert the target bounding boxes of each note contained in each segment image into the start time, end time and pitch of the corresponding note, and to perform time splicing on the start time, end time and pitch of each note contained in each segment image to obtain the transcription result of the audio to be transcribed. Specifically, based on the transcription model and combined with the prior bounding boxes of each note in the aligned reference score, the target bounding boxes of each note contained in each slice image are predicted, including: The encoder based on the transcription model extracts audio features from each segment of the image to obtain the audio feature map of the corresponding segment image, and uses the prediction head to map the audio feature map of the corresponding segment image into prediction boxes of multiple notes; wherein, the prior localization box and the prediction box include the starting position coordinates of the note, the duration of the note and the frequency. The mapping relationship between the prior localization box of each note in the aligned reference score and the prediction box corresponding to each segment image is determined, and the difference between the prediction box corresponding to each segment image and its mapped prior localization box is calculated as the uncertainty of the corresponding prediction box. For each segmented image, based on the uncertainty of the prediction bounding box corresponding to the corresponding segmented image, the K prediction bounding boxes with the smallest uncertainty are selected to correspond to the audio features in the audio feature map of the corresponding segmented image, so as to construct the initial query vector of the corresponding segmented image. Based on the decoder of the transcription model, the initial query vector of the corresponding segmented image is used to perform attention transformation on the audio feature map of the corresponding segmented image. Based on the attention transformation result, the target bounding box of each note contained in the corresponding segmented image is predicted.

9. An electronic device, comprising: include: Computer-readable storage media and processors; The computer-readable storage medium is used to store executable instructions; The processor is configured to read executable instructions stored in the computer-readable storage medium and execute the automatic music transcription method based on a reference score as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer instructions for causing the processor to perform the automatic music transcription method based on the reference musical score according to any one of claims 1-7.

Images