A note-level automatic singing transcription method based on target detection and language features

Abstract

The application provides a note-level automatic singing transcription method based on target detection and language characteristics, comprising the following steps: step 1: converting a one-dimensional audio sequence into two-dimensional mel spectrum slice and phoneme posterior slice with similar width-height ratio through a preprocessing method of mel transformation, phoneme classification, linear intensity mapping and slicing; step 2: performing target detection on the mel spectrum slice and phoneme posterior slice, post-processing and time adjustment on the left and right boundaries of the boundary box obtained by target detection, and then obtaining the final start time and end time through decision screening; step 3: taking the lower boundary of the target detection boundary box as the fundamental frequency, obtaining the final fundamental frequency through peak value search, and then converting the final fundamental frequency to obtain the MIDI pitch value. The method can effectively improve the phoneme feature extraction effect, improve the phoneme posterior slice quality and improve the feature extraction and analysis effect, thereby improving the transcription accuracy.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of singing transcription, and particularly relates to a note-level automatic singing transcription method based on target detection and language features. BACKGROUND

[0002] Note-level automatic singing transcription is one of the most representative music information retrieval tasks, which can predict the attributes of notes, i.e., onset, offset and pitch, from singing audio. Although the research on note-level singing transcription has been for quite a long time, the effect of singing transcription is usually not as good as that of single-instrument transcription such as piano, and the reason is that singing has highly diversified expressiveness in pitch, timbre, dynamics and pronunciation of lyrics. For example, singing techniques such as vibrato and slide make it difficult to find note boundaries, and the lack of large-scale annotated datasets is one of the main obstacles of this technology.

[0003] There are mainly three kinds of automatic singing transcription (AST) methods at present. The first kind of method adopts an end-to-end way to obtain frame-level onset, offset time and pitch probability, and then obtains note-level results through post-processing. For example, a convolutional neural network (CNN) is first used to extract high-dimensional features of a spectrogram, and then a fully connected layer is used to obtain frame-level transcription results. The second kind of method first obtains onset and offset time, and then obtains pitch through pitch extraction. For example, correntropy is first used to detect onset and offset time, and then a standard pitch tracking algorithm is used to obtain pitch. The third kind of method first performs fundamental frequency (F0) tracking to obtain a frame-level pitch curve, and then performs post-processing on the pitch curve to obtain note-level onset, offset time and pitch. For example, the Tony software first performs pitch tracking through the pYIN algorithm. Then, a Viterbi decoding algorithm based on a hidden Markov model (HMM) is used to obtain note-level transcription results.

[0004] But the above method for starting and ending time detection is mostly based on frame-level transcription, and frame-level detection cannot remove the frequency fluctuations caused by singing methods such as glissando and vibrato, which leads to false detection problems. In view of this limitation, a target detection-based method appears, which considers the features of the entire note in the spectrogram instead of the transient frame features near the start and end points, so that the start and end points can be observed from a more macroscopic perspective, reducing the impact of frequency fluctuations. On the ISMIR2014 dataset, an F1 detection score of more than 90% for the start and end points is obtained. However, the target detection-based method still has obvious defects. Since the method is based on target detection of spectrogram features, it may miss the second note when there is no obvious interval between two consecutive notes, which is called continuous missing. This type of false detection reduces the detection accuracy to some extent. In actual testing, some slow and emotional songs may also have obvious transcription errors. SUMMARY

[0005] In order to overcome the shortcomings of the prior art, the purpose of the present application is to provide a note-level automatic singing transcription method based on target detection and language features. The method converts a one-dimensional audio sequence into a mel-spectrogram through preprocessing, which is the main input of the target detection model. Then, the mel-spectrogram is input into the designed phoneme classifier to return the phoneme posterior graph (PPG). The mel-spectrogram and the phoneme posterior graph are input into the target detection model YOLOX, and the boundary box is detected in a macroscopic perspective by slicing. The preliminary start and end time sequences and pitch value sequences are obtained through post-processing and decision screening to obtain the final start and end time and pitch value. This method can effectively improve the phoneme feature extraction effect, improve the quality of the phoneme posterior graph, and improve the feature extraction and analysis effect, thereby improving the transcription accuracy. In order to achieve the above purpose, the technical scheme adopted by the present application is:

[0006] A note-level automatic singing transcription method based on target detection and language features, comprising the following steps:

[0007] Step 1: Convert a one-dimensional audio sequence into a two-dimensional mel-spectrogram slice with a similar aspect ratio (approximately square) and a phoneme posterior graph slice.

[0008] Step 2: Perform target detection on the mel-spectrogram and phoneme posterior graph slices, and perform post-processing and time adjustment on the left and right boundaries of the target detection boundary box. Then, the final start time and end time are obtained through decision screening.

[0009] Step 3: Take the lower boundary of the target detection boundary box as the fundamental frequency, and obtain the final fundamental frequency through peak value search. Then, the final fundamental frequency is converted to obtain the MIDI pitch value, and the automatic singing transcription is completed.

[0010] The step 1 is specifically:

[0011] 1.1 Time-frequency conversion and phoneme classification

[0012] The mel-spectrogram is used as the main input, and the phoneme posterior is used as the second input to extract note features;

[0013] The Mel transform is used to convert the audio into a corresponding mel-spectrogram, which is one of the main inputs of the target detection model. Then, the mel-spectrogram is used as the input of the phoneme classifier to extract language information and obtain the corresponding phoneme posterior;

[0014] 1.2 Linear intensity mapping

[0015] Linear intensity mapping is performed on the mel-spectrogram and phoneme posterior to enrich and refine the information contained in the image;

[0016] Linear intensity mapping converts a single-channel mel-spectrogram into a three-channel RGB image. Linear intensity mapping effectively quantizes the original image through spectral intensity. The formula of linear intensity mapping is generally expressed as:

[0017] RGB(x,y)=(f R (s(x,y)),f G (s(x,y)),f B (S(x,y)))

[0018] Where S(x,y) is the intensity value of the single-channel spectrogram at position (x,y), f R , f G , and f B are functions for mapping intensity values to the three channels of RGB, which are defined here as:

[0019] f R (S)=a R ·S+b R

[0020] f G (S)=a G ·S+b G

[0021] f B (S)=a B .S+b B

[0022] Where a R , a G , a B are proportional coefficients, and b R , b G , b B are offset values;

[0023] 1.3 Slice generation

[0024] After converting the audio to a mel-spectrogram and phoneme posteriorgram and linear intensity mapping, an automatic slice generation algorithm is used to cut the image along the time axis into roughly square slices, using ratio to describe the slice shape.

[0025] The step 1.1 is specifically:

[0026] The phoneme classifier is based on a ConvNet architecture, using a connectionist temporal classification (CTC) loss function that handles the alignment between the predicted phoneme sequence and the real phoneme sequence of unequal length;

[0027] On the basis of the original ConvNet structure, i.e. after the fully connected layer, two layers of bidirectional LSTM layers and a fully connected layer are added to reconstruct the log-mel spectrogram, and the two layers of bidirectional LSTM structure can extract and integrate the time sequence features in a deeper level on the basis of the preliminarily generated mel spectrogram, and the fully connected layer maps the high-dimensional LSTM output back to the dimension of the original input, i.e. the reconstructed log-mel spectrogram so as to enhance the time alignment when using the CTC loss, and when calculating the reconstruction loss , first, the mel spectrogram on the logarithmic scale is normalized to make its value between -1 and 1, and then the tanh activation function is used in the corresponding layer of the neural network to ensure that the range of the output value is consistent with the range of the normalized value, and the definitions of the loss functions are as follows:

[0028]

[0029] wherein and are the connectionist temporal classification loss and the total output loss of the phoneme posteriorgram respectively, T is the total number of time steps, B is a function that maps the predicted phoneme sequence to the real phoneme sequence, is the original input mel spectrogram, p is the real phoneme sequence, is the phoneme posteriorgram (PPG) at time t.

[0030] The calculation formula in the step 1.3 is as follows:

[0031] ratio=w / h-1

[0032] wherein w and h represent the width and height of the slice, and the calculation formula is as follows:

[0033]

[0034] where frame len is the frame length, n bins is the frequency band, and scale is the scale factor from matrix size to image size.

[0035] Split the audio to get the segment S using librosa.effect.split function i i = 1, 2, 3, …, M, return the start time and end time, then calculate the aspect ratio ratio according to the width and height of the image respectively, and obtain the corresponding split point, and continuously iterate and optimize the split point group splits until the maximum iteration number M is reached.

[0036] The step 2 is specifically:

[0037] 2.1 Note target detection

[0038] Draw a bounding box on the mel spectrogram slice obtained in step 1, and annotate the left and right boundaries of the phoneme posterior graph slice based on the original start and end time of the audio when constructing the data set for phoneme target detection using labelme software. Then input the annotated segment into the target detection model for training.

[0039] 2.2 Bounding box post-processing and time adjustment

[0040] The detected bounding box still uses the original processing logic based on the target detection transcription framework, which can be summarized as two steps: removing nested bounding boxes and creating larger bounding boxes, and ensuring that adjacent bounding boxes are not intersected.

[0041] if x b1 ≤x a1 and y b1 ≤y a1 and x a2 ≤x b2 and y a2 ≤y b2 :

[0042] C = (min(x a1 , x b1 ), min(y a1 , y b1 ), max(x a2 , x b2 ), max(y a2 , y b2 ))

[0043] boxes.remove(A) and boxes.remove(B)

[0044] boxes.append(C)

[0045] For each pair of bounding boxes (A, B), the coordinate of the top-left corner of bounding box A is (x...). a1 ,y a1 The coordinates of the lower right corner are (x a2 ,y a2 The top-left corner coordinate of bounding box B is (x b1 ,y b1 The coordinates of the lower right corner are (x b2 ,y b2 Check if the nested bounding box condition is met. If bounding box A is contained in bounding box B or bounding box B is contained in bounding box A, then create a new bounding box C and replace bounding box A and bounding box B.

[0046] if x a2 >x b1 And y a1 ≤y b2 And y a2 ≥y b1 :

[0047] x b1 =x a2

[0048] For each pair of bounding boxes (A, B), if they intersect, adjust the left boundary x of bounding box B. b1 =x a2 .

[0049] Based on the relationship between the slice image size and duration, the start time, end time, and fundamental frequency of each slice image are obtained. The start time corresponds to the left boundary of the bounding box, the end time corresponds to the right boundary of the bounding box, and the fundamental frequency corresponds to the bottom boundary of the bounding box.

[0050] The start and end times of the initial complete audio are obtained by adjusting the start and end times. Since the spectrogram and phoneme posterior map have been sliced ​​before, all image slices are now stitched together along the time axis, and the corresponding time offsets are added to the previously detected start and end times.

[0051] 2.3 Start and End Time Decisions

[0052] The Mel spectrogram and phoneme posterior map are input into the target detection model and post-processed to obtain the corresponding initial complete audio start and end times. After screening and decision-making, the final start and end times are determined.

[0053] In step 2.1:

[0054] YOLOX is used as the target detection model, and its loss function is a comprehensive multi-task loss function, which includes three main parts: classification loss, localization loss, and objectness loss. The loss function of YOLOX can be expressed as the following formula:

[0055] Loss = λ cls · Loss cls + λ loc · Loss loc + λ obj · Loss obj

[0056] where Loss cls is the classification loss, which measures the difference between the predicted class and the true class, Loss loc is the localization loss, which measures the difference between the predicted bounding box and the true bounding box, and Loss obj is the objectness loss, which measures the confidence error of whether there is a target. λ cls , λ loc , and λ obj are the weight coefficients corresponding to the loss terms.

[0057] In step 2.3:

[0058] The two sets of starting point time information generated based on the spectrogram and the phoneme posterior graph are compared one by one. The overlapping starting point information is directly classified as one of the final outputs, and the two sets of data continue to be compared one by one. The non-overlapping starting point, i.e., one party has a starting point t i while the other party has no time information at the corresponding time point.

[0059] According to whether the time difference of the non-overlapping starting point is within the tolerance time, it can be divided into two cases: one is that there is time information within the time tolerance range of the corresponding time point, which can be considered as overlapping, and t i is taken as one of the final outputs, and the two sets of data continue to be compared one by one. The second is that there is no time information within the time tolerance range of the corresponding time point, which means that there is a detection error or at least one note is missed. At this time, the starting point t i is taken as one of the final outputs, and the data set with the starting point t i moves forward by one starting point t i+1 , and the data set without the starting point t i does not move forward and waits for comparison with t i+1 . Until all the starting points are compared, the final output of the starting point is obtained, and the decision logic of the end point is the same as that of the starting point.

[0060] The decision function formula can be expressed as:

[0061] S3=collect_start_points(S1,S2)

[0062] E3=collect_end_points(E1,E2)

[0063]

[0064] Wherein S1, S2, E1, E2 respectively represent the initial start point, end point set of the spectrum graph and the phoneme posterior graph output, S3, E3 respectively represent the final output set of the start point, end point, t1∈S1, E1, t2∈S2, E2, t t Indicates the tolerance time.

[0065] The step 3 is specifically:

[0066] The lower boundary of the spectrum graph detection boundary box is used as the initial fundamental frequency to obtain the pitch, and the MIDI value (p) is used to represent the fundamental frequency (f), and the conversion relationship between p and f is as follows:

[0067]

[0068] According to the start and end time of the detected musical note, the fundamental frequency parameter, the frequency search range is determined, the local maximum value of the spectrum in each time frame is found, and the obtained peak frequency is accumulated, then the final accurate fundamental frequency of each musical note is calculated by averaging these peak frequencies, and is converted into a standard pitch MIDI value, and finally the pitch list of all musical notes is output.

[0069]

[0070] f pitch Indicates the final fundamental frequency, and f indicates the initial fundamental frequency corresponding to the lower boundary of the boundary box. Thus, the final output list of the start time, the end time and the pitch is obtained.

[0071] The beneficial effects of the present application are:

[0072] The phoneme classifier architecture in the present application combines the ConvNet structure with the bidirectional LSTM, compared with the traditional single acoustic model, can effectively improve the phoneme feature extraction effect and improve the phoneme posterior graph quality;Through linear intensity mapping, the information contained in the mel spectrum graph and the phoneme posterior graph can be enriched and detailed, the feature extraction and analysis effect is improved, so as to improve the transcription accuracy;The mel spectrum graph and the phoneme posterior graph with too large aspect ratio are automatically sliced into slices with similar aspect ratio and approximately square, which is easy to be processed by the target detection model, and the transcription speed is effectively improved.

[0073] Compared with other target detection models, the YOLOX target detection model has the advantages of single-stage processing and anchor-free training, effectively reduces the potential risk of overfitting in the detection process, and improves the target detection accuracy; the initial start and end time obtained by the decision function is screened and decided, compared with the existing detection of the start and end time of the musical note, the error caused by the transcription result obtained by a single feature can be effectively reduced, and the transcription accuracy of the start and end time of the musical note is improved.

[0074] In the present application, the initial fundamental frequency is obtained by using the peak search method with the lower boundary of the boundary box, compared with the method of obtaining the pitch by signal processing in the past, the frequency of the actual musical note can be more accurately captured, the transcription error is reduced, the calculation complexity is effectively reduced, and the transcription speed is improved.

[0075] In summary, the present application proposes a musical note level automatic singing transcription method based on target detection and language features to solve the problem of low transcription accuracy caused by continuous omission and other transcription errors in the current automatic singing transcription. The mel-spectrogram is used as the main input feature, the factor classifier is designed to obtain the phoneme posterior graph as the second input feature, then the two are linearly intensity mapped, the image information is enriched, and then automatic slicing processing is performed, the obtained slices enter the target detection model to obtain the detection boundary box, the boundary boxes of each slice are post-processed, spliced and time adjusted to obtain the initial start and end time and the fundamental frequency of the complete audio, the initial and end time obtained by the spectrum graph and the phoneme posterior graph are screened and decided by using the decision function to obtain the final initial and end time, the lower boundary of the boundary box is used as the initial fundamental frequency, and the peak search method is used to obtain the final pitch value, so as to obtain the final musical note information, and complete the automatic singing transcription. The present application proposes a complete automatic singing transcription framework, captures the time and frequency characteristics of the actual musical note, reduces the transcription error of continuous omission, reduces the calculation complexity, and improves the accuracy of automatic singing transcription. BRIEF DESCRIPTION OF DRAWINGS

[0076] Figure 1 It is a display comparison diagram of continuous musical notes in the spectrum graph, the piano scroll and the phoneme posterior graph in the present application.

[0077] Figure 2 It is a schematic diagram of the factor classifier architecture designed in the present application.

[0078] Figure 3 It is a schematic diagram of the mel-spectrogram output by the input audio in the present application after Mel transformation.

[0079] Figure 4 It is a schematic diagram of the phoneme posterior graph output by the input audio in the present application after phoneme classifier.

[0080] Figure 5A comparison diagram of the mel-spectrogram before and after linear intensity mapping in the present application.

[0081] Figure 6 A diagram of the target detection result bounding box with the mel-spectrogram as the input feature and the target detection result bounding box with the phoneme posterior graph as the input feature in the present application.

[0082] Figure 7 A comparison fold line diagram of the start time set obtained by taking the mel-spectrogram and the phoneme posterior graph as the extracted features, the final decision output start time set, and the real start time set in the present application.

[0083] Figure 8 A comparison diagram of the transcription effect of the same fast-paced song with the mel-spectrogram as the input and with the mel-spectrogram and the phoneme posterior graph as the input in the present application.

[0084] Figure 9 A comparison diagram of the transcription effect of the same slow-paced song with the mel-spectrogram as the input and with the mel-spectrogram and the phoneme posterior graph as the input in the present application. DETAILED DESCRIPTION

[0085] The present application will be further described in detail below with reference to the accompanying drawings.

[0086] For the problem of continuous omission, the present application proposes a note-level automatic singing transcription method based on target detection and language features. The method converts a one-dimensional audio sequence into a mel-spectrogram through preprocessing, which is used as the main input of the target detection model. Then, the mel-spectrogram is input into the designed phoneme classifier to return the phoneme posterior graph (PPG), and both of them are used as the input of the target detection model YOLO. Through slicing, the bounding box is detected from a macro perspective. The preliminary start and end time sequences and the pitch value sequence are obtained through post-processing and are screened by decision to obtain the final start and end time and pitch value.

[0087] The reason for taking the phoneme as one of the language features as input is that singing conveys language information through lyrics, which will affect the segmentation of notes. As shown in the accompanying Figure 1 The spectrum graph, the piano roll with note start and end time and pitch, and the phoneme posterior graph of the singing are shown from top to bottom. It can be seen that at 1.7 seconds, the corresponding note cannot be segmented according to the mel-spectrogram alone, but the language feature in the phoneme posterior graph can better detect the starting point at the note transition, thereby reducing the occurrence of continuous omission errors.

[0088] A note-level automatic singing transcription method based on target detection and language features, comprising the following steps:

[0089] 1. Preprocessing

[0090] 1.1 Time-frequency conversion and phoneme classification

[0091] The previous algorithm mainly uses the spectrogram obtained by Mel transform, constant Q transform, etc. as input, and takes the spectrogram as the basis for feature extraction. This method has good feature extraction effect on the audio of instruments such as piano, but it is relatively difficult to achieve ideal effect for starting detection of singing voice by using only the spectral difference in the spectrogram. Therefore, the mel spectrogram is used as the main input, and the phoneme posterior graph is used as the second input to extract note features.

[0092] The audio is converted into a corresponding mel spectrogram by Mel transform, which is one of the main inputs of the target detection model. Then the mel spectrogram is taken as the input of the phoneme classifier to extract language information and obtain the corresponding phoneme posterior graph. The phoneme classifier uses ConvNet as the basic structure and uses the connectionist temporal classification (CTC) loss function. This function can handle the alignment between the predicted phoneme sequence and the real phoneme sequence with different lengths. However, since the CTC loss function is optimized for predicting the entire sequence, its prediction method is often sharp and sparse, so it cannot find the boundaries of factors well. To solve this problem, two layers of bidirectional LSTM layers and a fully connected layer are added after the original ConvNet structure, i.e. after the fully connected layer, to reconstruct the log mel spectrogram, thereby enhancing the time alignment when using the CTC loss. When calculating the reconstruction loss , the log-scale mel spectrogram is first normalized to have a value between -1 and 1. Then, the tanh activation function is used in the corresponding layer of the neural network to ensure that the range of output values is consistent with the range of normalized values. The definitions of the various loss functions are as follows:

[0093]

[0094] where and are the connectionist temporal classification loss and the total output loss of the phoneme posterior graph, respectively, T is the total number of time steps, B is a function that maps the predicted phoneme sequence to the real phoneme sequence, is the original input mel spectrogram, p is the real phoneme sequence, is the time phoneme posterior graph (PPG).

[0095] The designed phoneme classifier architecture is shown in the accompanying Figure 2 :

[0096] The mel spectrogram obtained by Mel transform of the input audio segment is shown in the accompanying Figure 3As shown, the vertical axis represents the Mel frequency of the input audio segment, and the color represents the power intensity at a specific time and frequency point; the phoneme posterior map output by the phoneme classifier for the same input audio is shown in the appendix. Figure 4 As shown, the vertical axis represents the types of phonemes, and the color represents the probability of a phoneme at a specific time.

[0097] 1.2 Linear Intensity Mapping

[0098] Linear intensity mapping was performed on both the spectrogram and the phoneme posterior map. However, since the masking operation was used when obtaining the phoneme posterior map to make the corresponding phoneme features clear enough, and the linear intensity mapping of the phoneme posterior map did not improve the extraction effect, the analysis mainly focuses on the effect of linear intensity mapping on the spectrogram.

[0099] Linear intensity mapping (LINE) transforms a single-channel spectrogram into a three-channel RGB image. LINE effectively quantizes the original image by spectral intensity. Under this mapping, strong spectral values ​​are more prominent in one channel, while weaker spectral values ​​are more prominent in other channels. This makes the image contain richer and more detailed information, thereby improving the effectiveness of feature extraction and analysis. The formula for LINE can generally be expressed as:

[0100] RGB(x,y)=(f R (s(x,y)),f G (s(x,y)),f B (S(x,y)))

[0101] Where S(x,y) is the intensity value of the single-channel spectrogram at position (x,y), f R f G and f B This is a function used to map intensity values ​​to the three RGB channels. Here, the functions are defined as follows:

[0102] f R (S)=a R ·S+b R

[0103] f G (S)=a G .S+b G

[0104] f B (S)=a B .S+b B

[0105] Where a R a G a B It is the proportionality coefficient, b R bG , b B Offset value.

[0106] As shown in the accompanying drawings, Fig. a) represents the mel-spectrogram before linear intensity mapping, and Fig. b) represents the mel-spectrogram after linear intensity mapping. Comparing Figs. a) and b), it can be seen that the mel-spectrogram after linear intensity mapping can contain more abundant and detailed information, which helps to improve the target detection accuracy and thus improve the transcription accuracy. Figure 5 1.3 Slice generation

[0107] After converting the audio into the corresponding image and performing linear intensity mapping, the image is too large in aspect ratio and is not convenient to process. Therefore, an automatic slice generation algorithm is used to cut the image into roughly square slices along the time axis. The ratio is used to describe the shape of the slice, and the calculation formula is as follows:

[0108] ratio = w / h - 1

[0109] where w and h represent the width and height of the slice, and the calculation formula is as follows:

[0110]

[0111]

[0112] where frame_len is the frame length, n_bins is the frequency band, and scale is the scale factor from the matrix size to the image size.

[0113] The librosa.effect.split function is used to split the audio to obtain the segment S i , i = 1, 2, 3,..., M, and the start time and end time are returned. Then, the aspect ratio ratio of the image width and height is calculated, and the corresponding split point is obtained, and the split point group splits is continuously iterated and optimized until the maximum iteration number M is reached.

[0114] 2. Note detection

[0115] 2.1 Note target detection

[0116] ​The automatic singing transcription framework based on target detection draws a bounding box on the overall features of the note object in the spectrogram to obtain the start point, end point and pitch. This algorithm draws a bounding box on the main input mel-spectrogram in the same way. For the input phoneme posterior graph, when constructing the data set for phoneme target detection, the left and right boundary annotations of the phoneme posterior graph are made based on the original start and end time annotations using the labelme software. The annotation of the pitch can be ignored because the vertical coordinates of the phoneme posterior graph correspond to each phoneme rather than the frequency, so the upper and lower boundaries coincide with the upper and lower boundaries of the phoneme posterior graph. Then, the annotated segments are input into the target detection model for training.

[0117] YOLOX is used as the target detection model. As an advanced target detection model, its loss function is a comprehensive multi-task loss function, which includes three main parts: classification loss, localization loss and objectness loss. The loss function of YOLOX can be expressed as the following formula:

[0118] Loss=λ cls ·Loss cls +λ loc ·Loss loc +λ obj ·Loss obj

[0119] where Loss cls is the classification loss, which measures the difference between the predicted class and the true class, Loss loc is the localization loss, which measures the difference between the predicted bounding box and the true bounding box, and Loss obj is the objectness loss, which measures the confidence error of whether there is a target. λ cls , λ loc , λ obj are the weight coefficients corresponding to the loss terms.

[0120] The YOLO-s pre-trained model is used as the initial model parameters, the number of target categories is set to 1, the input image size is 640x640, the learning rate, optimizer, etc. are the same as the original YOLOX model, the maximum number of training rounds is set to 300, and the confidence threshold is set to 0.6 and the non-maximum suppression is set to 0.45 in the inference stage.

[0121] As shown in the accompanying Figure 6As shown, Figure a) represents the target detection result bounding box with mel-spectrogram as input feature, the left boundary of the bounding box corresponds to the onset time of the note, the right boundary corresponds to the offset time of the note, and the lower boundary corresponds to the fundamental frequency of the note; Figure b) represents the target detection result bounding box with phoneme posterior as input feature, the left boundary of the bounding box corresponds to the onset time of the note, the right boundary corresponds to the offset time of the note.

[0122] 2.2 Bounding box post-processing and time adjustment

[0123] In target detection, especially for monophonic audio signals similar to human voice, the problem of nested bounding boxes must be handled and ensure that adjacent bounding boxes are disjoint. The detected bounding boxes still use the original processing logic based on the target detection transcription framework, which can be summarized as two steps: removing nested bounding boxes and creating larger bounding boxes, and ensuring that adjacent bounding boxes are disjoint:

[0124] if x b1 ≤x a1 and y b1 ≤y a1 and x a2 ≤x b2 and y a2 ≤y b2 :

[0125] C=(min(x a1 ,x b1 ),min(y a1 ,y b1 ),max(x a2 ,x b2 ),max(y a2 ,y b2 ))

[0126] boxes.remove(A))and boxes.remove(B)

[0127] boxes.append(C)

[0128] For each pair of bounding boxes (A, B), where the top-left corner coordinates of bounding box A are (x a1 ,y a1 ), the bottom-right corner coordinates are (x a2 ,y a2 ), the top-left corner coordinates of bounding box B are (x b1 ,y b1 ), and the bottom-right corner coordinates are (x b2 ,y b2 ), check if the nested bounding box condition is met. If bounding box A is contained by bounding box B or bounding box B is contained by bounding box A, create a new bounding box C and replace bounding box A and bounding box B.

[0129] if x a2 >x b1 and y a1 ≤y b2 and y a2 ≥y b1 :

[0130] x b1 =x a2

[0131] For each pair of bounding boxes (A, B), if intersecting, adjust the left boundary x b1 =x a2 of bounding box B.

[0132] The start and end values of the complete audio are obtained by adjusting the start and end values. Since the spectrogram and the phoneme posterior have been cut previously, all image slices are now spliced along the time axis, and the corresponding time offset is added to the previously detected start and end values.

[0133] 2.3 Start and end time decision

[0134] As mentioned above, the singing transcription framework based on target detection, although considering the overall features of the notes, still detects the start and end points based on the features extracted from the spectrogram, which cannot avoid the situation of continuous missing false detection. The automatic singing transcription algorithm based on target detection and language features proposed in this paper takes phonemes as part of the extracted features according to the characteristics of singing voice, which can effectively reduce continuous missing.

[0135] After inputting the mel-spectrogram and phoneme posterior into the target detection model and post-processing, the corresponding start and end times can be obtained, and the final start and end times are determined after screening decision.

[0136] Take the start time as an example: compare the two groups of start point time information generated based on the spectrogram and phoneme posterior one by one. The overlapping start point information is directly classified as one of the final outputs, and the two groups of data continue to be compared one by one. The non-overlapping start point, i.e., one party has a start point t i while the other party has no time information at the corresponding time point, can be divided into two cases: one is that there is time information within the time tolerance range left and right of the corresponding time point, which can be regarded as overlapping, and t i is taken as one of the final outputs, and the two groups of data continue to be compared one by one; the other is that there is no time information within the time tolerance range left and right of the corresponding time point, which means that there is a missing detection or at least one note error in the detection of the two parties, and the start point t i is taken as one of the final outputs, and the data group with the start point t i moves forward by one start point t i+1 .i The data set of S1 is not advanced, waiting for t i+1 The final output of the start point is obtained after comparing all the start points. The decision logic of the end point is the same as that of the start point.

[0137] The decision function formula can be expressed as:

[0138] S3 = collect_start_points (S1, S2)

[0139] E3 = collect_end_points (E1, E2)

[0140]

[0141] Wherein S1, S2, E1, E2 represent the initial start point and end point set output by the spectrum graph and phoneme posterior graph respectively, S3, E3 represent the final output set of start point and end point respectively, t1∈S1, E1, t2∈S2, E2, t t The tolerance time is represented.

[0142] As shown in the accompanying Figure 7 The mel_start, ppg_start, final_start and truth_start represent the spectrum graph, phoneme posterior graph, final output and true start point set respectively. The vertical coordinate represents the start point time. 1.71 seconds and 3.41 seconds represent the case that the spectrum graph and phoneme posterior graph do not detect the start point. The final output after decision can be consistent with the true start point set within the time tolerance.

[0143] 3. Pitch detection

[0144] Pitch is usually represented by fundamental frequency. The vertical coordinate of phoneme posterior graph corresponds to each phoneme rather than frequency, and cannot obtain pitch by target detection on phoneme posterior graph. Therefore, the lower boundary of the frame detected by the spectrum graph is used to obtain the pitch. For convenience, the MIDI value (p) is commonly used to represent the fundamental frequency (f), and the conversion relationship between p and f is as follows:

[0145]

[0146] According to the detected start and end time, fundamental frequency and other parameters of the note, the frequency search range is determined. In each time frame, the local maximum value of the spectrum is found, and these peak frequencies are accumulated. Then, the final accurate fundamental frequency of each note is calculated by averaging these peak frequencies, and it is converted into a standard pitch value. Finally, the pitch list of all notes is output.

[0147]

[0148] fpitch represents the final fundamental frequency, f represents the fundamental frequency corresponding to the lower boundary of the bounding box.

[0149] The final output list of start time, end time and pitch is obtained. The automatic singing transcription algorithm based on target detection and language features can achieve results comparable to the original transcription framework in songs with fast and clear rhythm and crisp pronunciation, and can avoid continuous missed detection in songs with slower tempo, slides and extended notes. The model outputs a list of note information in csv format, and two different styles of songs are selected for testing. The obtained note information is visualized, and Figure 8 represents the output results of fast rhythm songs in two models, Figure 9 represents the output results of slow songs in two models.

Claims

1. A note-level automatic singing transcription method based on target detection and language features, characterized in that, Comprising the following steps; Step 1: converting one-dimensional audio sequence into two-dimensional mel-spectrogram slices and phoneme posterior slices with similar aspect ratio, square shape; Step 2: target detection on mel-spectrogram slices and phoneme posterior slices, post-processing and time adjustment on left and right boundaries of the bounding boxes obtained by target detection, and finally obtaining the starting time and ending time through decision screening; Step 3: taking the lower boundary of the target detection bounding box as the fundamental frequency, obtaining the final fundamental frequency through peak value search, and then converting the final fundamental frequency to obtain the MIDI pitch value, completing automatic singing transcription; The step 1 is specifically: 1.1 time-frequency conversion and phoneme classification: Mel transform is used to convert audio into corresponding mel-spectrogram, which is used as the input of the target detection model, and then the mel-spectrogram is used as the input of the phoneme classifier to extract language information and obtain the corresponding phoneme posterior graph; the corresponding phoneme posterior graph is input into the target detection model; 1.2 linear intensity mapping: Linear intensity mapping converts single-channel mel-spectrogram into three-channel RGB image, which is represented as: RGB(x,y) = (f R (s(x,y)), f G (s(x,y)), f B (S(x,y))) where S(x, y) is the intensity value of the single-channel spectrogram at position (x, y), f R , f G , and f B are functions for mapping intensity values to the RGB three channels, defined as: f R (S) = a R • S + b R f G (S) = a G • S + b G f B (S) = a B • S + b B where a R , a G , a B are proportional coefficients, b R , b G , b B are offset values; 1.3 slice generation: The automatic slice generation algorithm is used to cut the image into square slices along the time axis, and the ratio is used to describe the slice shape.

2. The note-level automatic singing transcription method based on target detection and language features according to claim 1, characterized in that, The step 1.1 is specifically: The phoneme classifier is based on a ConvNet architecture using a connectionist temporal classification (CTC) loss function that handles the alignment between the predicted phoneme sequence and the real phoneme sequence of varying length; On the basis of the original ConvNet structure, that is, adding two layers of bidirectional LSTM layers and a fully connected layer after the fully connected layer to reconstruct the log mel-spectrogram; The two-layer bidirectional LSTM structure further extracts and integrates the time sequence features on the basis of the preliminarily generated mel-spectrogram; The fully connected layer maps the high-dimensional LSTM output back to the dimension of the original input, i.e., the reconstructed log-mel spectrogram Thus enhancing the time alignment when using CTC loss; When calculating the reconstruction loss , the log-scale mel spectrogram is first normalized to have values between -1 and 1, and then a tanh activation function is used in the corresponding layer of the neural network to ensure that the range of output values is consistent with the range of normalized values. The definitions of the individual loss functions are as follows: wherein and are the connectionist temporal classification loss, the total output loss of the phoneme posterior graphs, respectively, T is the total number of time steps, B is a function that maps the predicted phoneme sequence to the true phoneme sequence, is the original input mel-spectrogram, p is the true phoneme sequence, is the phoneme posterior graph (PPG) at time t.

3. The note-level automatic singing transcription method based on target detection and language features according to claim 1, characterized in that, The calculation formula in the step 1.3 is as follows: ratio=w / h-1 Where w and h represent the width and height of the slice, and the calculation formula is as follows: Where frame_len is the frame length, n_bins is the frequency band, and scale is the scale factor of the matrix size to the image size; Split the audio to get segments S using librosa.effect.split function i i = 1, 2, 3, …, M, return the start time and end time, then calculate the aspect ratio ratio according to the width and height of the image respectively, and obtain the corresponding split point, and continuously iterate and optimize the split point group splits until the maximum iteration number M is reached.

4. The note-level automatic singing transcription method based on target detection and language features according to claim 1, characterized in that, The step 2 is specifically: 2.1 note target detection: Draw a bounding box on the mel-spectrogram slice obtained in step 1, and for the phoneme posterior slice, when constructing the data set for phoneme target detection, use the labelme software to annotate the left and right boundaries of the phoneme posterior slice based on the original start and end time of the audio, and then input the annotated segment into the target detection model for training; 2.2 boundary box post-processing and time adjustment: The detected boundary box adopts the processing logic based on the target detection transcription framework, which removes the nested boundary box and creates a larger boundary box, and ensures that adjacent boundary boxes are not intersected: if x b1 ≤x a1 and y b1 ≤y a1 and x a2 ≤x b2 and y a2 ≤y b2 : C = (min(x a1 ,x b1 ), min(y a1 ,y b1 ), max(x a2 ,x b2 ), max(y a2 ,y b2 )) boxes.remove(A)and boxes.remove(B) boxes.append(C) For each pair of bounding boxes (A, B), where the upper left corner coordinates of the bounding box A are the lower right corner coordinates are the upper left corner coordinates of the bounding box B are the lower right corner coordinates are check if the nested bounding box condition is met, if the bounding box A is contained by the bounding box B or the bounding box B is contained by the bounding box A, create a new bounding box C and replace the bounding box A and the bounding box B; if x a2 >x b1 and y a1 ≤y b2 and y a2 ≥y b1 : x b1 = x a2 For each pair of bounding boxes (A, B), if intersecting, adjust the left boundary of bounding box B According to the relationship between the size of the slice image and the duration, the starting time, the ending time and the fundamental frequency of each slice image are obtained, the starting time corresponds to the left boundary of the boundary box, the ending time corresponds to the right boundary of the boundary box, and the fundamental frequency corresponds to the lower boundary of the boundary box; By adjusting the starting time and the ending time, the starting time and the ending time of the initial complete audio are obtained, all image slices are spliced along the time axis, and the corresponding time offset is added to the previously detected starting time and ending time; 2.3 Start, end time decision: After inputting the mel-spectrogram and phoneme posterior into the target detection model and post-processing, the corresponding initial complete audio start time and end time are obtained, and the final start and end time are determined after screening decision.

5. The note-level automatic singing transcription method based on target detection and language features according to claim 4, characterized in that, In step 2.1: YOLOX is used as the target detection model, which includes three main parts: classification loss, localization loss, and objectness loss. The loss function of YOLOX is expressed as the following formula: Loss = λ cls • Loss cls + λ loc • Loss loc + λ obj • Loss obj where Loss cls is the classification loss measuring the difference between the predicted class and the true class, Loss loc is the localization loss measuring the difference between the predicted bounding box and the true bounding box, Loss obj is the target loss, λ cls , λ loc , λ obj are the weight coefficients for the corresponding loss terms.

6. The note-level automatic singing transcription method based on target detection and language features according to claim 4, characterized in that, In step 2.3: The two groups of starting point time information generated based on the spectrogram and the phoneme posterior graph are compared one by one. The coincident starting point information is directly classified into one of the final outputs, and the two groups of data continue to be compared one by one. The non-coincident starting point, that is, one party has a starting point t i while the other party has no time information at the corresponding time point. According to whether the time difference of the non-coincidence starting point is within the tolerance time, two cases are divided: one is that there is time information within the time tolerance range of the corresponding time point, which is regarded as a coincidence case, and t i As one of the final outputs, the two groups of data continue to be compared one by one in turn; Second, if there is no time information in the time tolerance range around the corresponding time point, it means that there is a missed detection or at least one note error in the detection of the two parties. At this time, the starting point t i is determined as the final output. i As one of the final outputs, there is a starting point t i+1 , and the data group advances one starting point t i , and the data group without the starting point t i+1 does not advance, waits to be compared with t i+1 , until all the starting points are compared, the final output of the starting point is obtained, and the decision logic of the ending point is the same as that of the starting point.

7. The note-level automatic singing transcription method based on target detection and language features according to claim 6, characterized in that, The decision function formula is expressed as: S3 = collect_start_points(S1, S2) E3 = collect_end_points(E1, E2) wherein S1, S2, E1, E2 represent initial start point, end point set of the spectrum and phoneme posterior output respectively, S3, E3 represent final output set of start point, end point respectively, t1∈S1, E1, t2∈S2, E2, t t represents the tolerance time.

8. The note-level automatic singing transcription method based on target detection and language features according to claim 4, characterized in that, The step 3 is specifically: The lower boundary of the bounding box detected by the spectrogram is used as the initial fundamental frequency to obtain the pitch, and the MIDI value (p) is used to represent the fundamental frequency (f). The conversion relationship between p and f is as follows: According to the detected start and end time of the note, the fundamental frequency parameter, the frequency search range is determined, the local maximum value of the spectrum is found in each time frame, and the obtained peak frequencies are accumulated. Then, the final accurate fundamental frequency of each note is calculated by averaging these peak frequencies, and it is converted into a standard pitch MIDI value. Finally, the pitch list of all notes is output. f pitch denotes the final fundamental frequency, f denotes the initial fundamental frequency corresponding to the lower boundary of the bounding box, and thus the final output list of start times, end times, and pitches is obtained.

Images