A multi-stage automatic pitch scoring method based on fragmentation

Abstract

This invention provides a segment-based, multi-stage automatic pitch scoring method in the field of singing evaluation technology, comprising the following steps: Step S10, acquiring audio data to be scored and sheet music data; Step S20, verifying the presence or absence of singing in the audio data; Step S30, performing noise reduction processing on the audio data using a deep learning model; Step S40, removing non-singing segments from the audio data; Step S50, detecting the note timing points of each note in the audio data; Step S60, extracting audio features from the audio data based on each note timing point, and organizing the sheet music data to obtain a sheet music file; Step S70, automatically scoring the audio data based on the sheet music file and audio features, generating and displaying the scoring results. The advantage of this invention is that it greatly improves the accuracy of singing pitch scoring.

Description

Technical Field

[0001] This invention relates to the field of singing evaluation technology, and in particular to a multi-stage automatic pitch scoring method based on a segment-based approach. Background Technology

[0002] As the importance of holistic education for primary and secondary school students is increasingly emphasized, music education is also receiving more and more attention. Music education itself possesses strong artistic and professional characteristics. When professionally grading students' humming audio recordings, the manual involvement of teachers is required. When large-scale grading is needed, it consumes a huge amount of human resources. At the same time, this traditional grading mechanism cannot provide students with real-time teaching feedback.

[0003] With the development of artificial intelligence technology, various music-assisted education systems have been developed and are playing a role in school education. However, most traditional music-assisted education systems are designed around entertainment, focusing primarily on the entertainment experience in practical applications. They cannot guarantee the accuracy of singing scores and cannot simulate the scoring level of a teacher. Therefore, they are not suitable for use in singing education for primary and secondary school students, especially in examinations, where inaccurate scoring can lead to unfairness.

[0004] Therefore, how to provide a segment-based, multi-stage automatic pitch scoring method to improve the accuracy of singing pitch scoring has become an urgent technical problem to be solved. Summary of the Invention

[0005] The technical problem to be solved by this invention is to provide a multi-stage automatic pitch scoring method based on segmentation, so as to improve the accuracy of singing pitch scoring.

[0006] This invention is implemented as follows: a multi-stage automatic pitch scoring method based on segmentation, comprising the following steps:

[0007] Step S10: Obtain the audio data and sheet music data to be scored;

[0008] Step S20: Verify whether there is singing in the audio data;

[0009] Step S30: Perform noise reduction processing on the audio data using a deep learning model;

[0010] Step S40: Remove the non-singing segments from the audio data;

[0011] Step S50: Detect the note timing of each note in the audio data;

[0012] Step S60: Extract audio features from the audio data based on the time points of each note, and organize the score data to obtain a score file;

[0013] Step S70: Automatically score the audio data based on the sheet music file and audio features, and generate and display the scoring results.

[0014] Further, step S10 specifically includes:

[0015] Obtain the audio data to be scored and the corresponding sheet music data.

[0016] Further, step S20 specifically includes:

[0017] The first frequency of the audio data is extracted using a Python dependency library. The extracted first frequency is converted into a first pitch value sequence. The length A of the first pitch value sequence and the length B of the first pitch value sequence after removing null values ​​are measured. It is determined whether B / A is greater than 0.21. If yes, it means that the audio data contains singing, and the process proceeds to step S30. If no, it means that the audio data does not contain singing, a score of 0 is generated and displayed, and the process ends.

[0018] Further, step S30 specifically includes:

[0019] After the audio data is denoised once using the Demucus model, it is then denoised a second time using the Spleeter model.

[0020] Further, step S40 specifically includes:

[0021] Step S41: Detect the first starting time point of the audio data using a deep learning model;

[0022] Step S42: Detect the second starting time point of the audio data using a traditional algorithm model;

[0023] Step S43: Based on the first and second starting times, cut a first audio segment from the audio data, extract the second frequency of the first audio segment using a Python dependency library, convert the extracted second frequency into a second pitch value sequence, measure the length 'a' of the second pitch value sequence, measure the length 'b' of the second pitch value sequence after removing null values, and determine whether 'b / a' is greater than 0.65. If so, use the smaller value between the first and second starting times as the third starting time; otherwise, use the larger value between the first and second starting times as the third starting time.

[0024] Step S44: Remove the non-singing segments from the audio data before the third starting time point.

[0025] Further, step S50 specifically includes:

[0026] Step S51: Using Python dependency libraries, slice the audio data according to the silence points in the audio time domain and convert it into a spectrum image;

[0027] Step S52: After performing second harmonic processing on each of the spectrum images, the abrupt change position and decay position of energy in the spectrum image are detected by the target detection model. The abrupt change position is taken as the starting position of the note, and the position with a value of 0 after the decay position is taken as the ending position of the note.

[0028] Step S53: Convert the start position and end position into the corresponding note time points;

[0029] Step S54: Detect the peak time point of the spectral energy of the audio data using the Python dependency library, and correct and supplement the note time points based on the peak time point of the spectral energy.

[0030] Further, step S60 specifically includes:

[0031] Based on the time points of each note, the audio data is sliced ​​to obtain several second audio segments. The third pitch value sequence of the second audio segments is extracted using a Python dependency library. The mode of each third pitch value sequence is used as the audio feature of the audio data.

[0032] The sheet music data is scanned using a sheet music recognition tool to identify the pitch value and time point corresponding to each character in the sheet music data, thereby obtaining a sheet music file in JSON format.

[0033] Further, step S70 specifically includes:

[0034] The audio features are subjected to pitch dilation based on the sampling rate of the audio format. Then, the pitch values ​​of the audio features and the score file are aligned using a dynamic time warping algorithm. Next, the pitch difference between the audio features and the score file is calculated. Based on the pitch difference, the pitch accuracy of the audio data is automatically scored, and the scoring results are generated and displayed.

[0035] The advantages of this invention are:

[0036] By verifying the presence or absence of singing in audio data, scoring audio data without singing is avoided, reducing the waste of computing resources. Noise reduction is performed on the audio data using a deep learning model, facilitating subsequent audio feature extraction. Non-singing segments are removed from the audio data to avoid scoring noise. The accuracy of note timing is effectively improved by detecting the note timing points in the audio data and correcting and supplementing them based on the peak time points of the spectral energy. Audio features are extracted from the note timing points, and the corresponding sheet music data is processed to obtain sheet music files. Finally, the audio data is automatically scored based on the sheet music files and audio features, generating and displaying the scoring results. This process involves quantifying audio features at a fine-grained note level, aligning the audio features with the sheet music files based on pitch values, calculating pitch differences, and then automatically scoring based on these differences. The scoring process combines machine learning, deep learning, traditional algorithms, relevant audio processing technologies, and audio feature extraction methods, ultimately greatly improving the accuracy of singing pitch scoring. Attached Figure Description

[0037] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0038] Figure 1 This is a flowchart of a multi-stage automatic pitch scoring method based on fragmentation according to the present invention.

[0039] Figure 2 This is a flowchart of the note timing detection of the present invention.

[0040] Figure 3 This is a flowchart of the note timing correction and supplementation of the present invention.

[0041] Figure 4 This is a flowchart of the audio data scoring process of this invention.

[0042] Figure 5 This is a flowchart of the starting point detection process of the present invention. Detailed Implementation

[0043] The overall approach of the technical solution in this application is as follows: By verifying the presence or absence of singing in the audio data, performing noise reduction, and removing non-singing segments, the influence of invalid information on the scoring is avoided; the note timing points (start and end times of notes) are corrected and supplemented by the peak time points of the spectral energy to improve the accuracy of note positioning; the audio data is divided into several small segments by the note timing points, then the audio features of the audio data are extracted, and the corresponding sheet music data is organized to obtain a sheet music file; finally, the pitch accuracy of the audio data is automatically scored based on the sheet music file and the audio features, that is, the audio features are quantified based on the fine granularity of the notes, and the pitch difference is calculated after aligning the pitch values ​​of the audio features and the sheet music file, and then the automatic scoring is performed based on the pitch difference, thereby improving the accuracy of the singing pitch scoring.

[0044] Please refer to Figures 1 to 5 As shown, a preferred embodiment of the present invention, a multi-stage automatic pitch scoring method based on fragmentation, includes the following steps:

[0045] Step S10: Obtain the audio data and sheet music data to be scored;

[0046] Step S20: Verify whether there is singing in the audio data;

[0047] Step S30: Perform noise reduction processing on the audio data using a deep learning model;

[0048] Step S40: Remove the non-singing segments from the audio data;

[0049] In an ideal recording environment, students should start singing when the "start singing" button is displayed, and should not speak when the "start singing" button is not displayed to avoid recording any noise. However, in practice, students' talking or humming sounds are often recorded before the singing segment begins. These sounds are actually noise outside the singing segment and should be cut out.

[0050] Step S50: Detect the note timing of each note in the audio data;

[0051] Note timing detection, also known as note start point detection, is a very important algorithm in music signal processing. The note start point refers to the moment when a key is pressed. A characteristic of the note start point is a sudden increase in energy or a change in the distribution of spectral energy.

[0052] Step S60: Extract audio features from the audio data based on the time points of each note, and organize the score data to obtain a score file; score based on the fine granularity of the notes to effectively improve the accuracy of the score;

[0053] Step S70: Automatically score the audio data based on the sheet music file and audio features, and generate and display the scoring results.

[0054] By automatically scoring the pitch accuracy of audio data and generating and displaying the scoring results, the real-time nature of the scoring is greatly improved, allowing students to understand their singing performance in a timely manner and adjust their learning accordingly.

[0055] Step S10 specifically involves:

[0056] The process involves acquiring the audio data to be scored and the corresponding sheet music data. The audio data can be obtained from various sources, such as music websites or manually collected data. This invention first collects audio data of primary and secondary school students singing, and then organizes the corresponding sheet music data based on the audio data as a reference for pitch scoring.

[0057] Step S20 specifically involves:

[0058] The first frequency of the audio data is extracted using a Python dependency library. The extracted first frequency is converted into a first pitch value sequence. The length A of the first pitch value sequence and the length B of the first pitch value sequence after removing null values ​​are measured. It is determined whether B / A is greater than 0.21. If yes, it means that the audio data contains singing, and the process proceeds to step S30. If no, it means that the audio data does not contain singing, a score of 0 is generated and displayed, and the process ends.

[0059] In practical applications, some recorded audio data contains no singing (i.e., some students turn on their microphones but do not actually sing). Therefore, these audio data should be filtered out before scoring to reduce the waste of computing resources.

[0060] Step S30 specifically involves:

[0061] After performing a first noise reduction process on the audio data using the Demucs model, a second noise reduction process is performed using the Spleeter model. Both the Demucs and Spleeter models are human voice separation models used to extract human voices from audio data, eliminating non-singing sounds such as accompaniment. This invention combines the Demucs and Spleeter models for noise reduction to obtain a pure human voice.

[0062] Because some noise is inevitably recorded during the recording process of singing audio, and the presence of this noise can have a significant impact on the accuracy of the subsequent scoring, noise reduction processing is necessary.

[0063] Step S40 specifically includes:

[0064] Step S41: Detect the first starting time point of the audio data using a deep learning model;

[0065] Step S42: Detect the second starting time point of the audio data using a traditional algorithm model;

[0066] Step S43: Based on the first and second starting times, cut a first audio segment from the audio data, extract the second frequency of the first audio segment using a Python dependency library, convert the extracted second frequency into a second pitch value sequence, measure the length 'a' of the second pitch value sequence, measure the length 'b' of the second pitch value sequence after removing null values, and determine whether 'b / a' is greater than 0.65. If so, use the smaller value between the first and second starting times as the third starting time; otherwise, use the larger value between the first and second starting times as the third starting time.

[0067] Step S44: Remove the non-singing segments from the audio data before the third starting time point.

[0068] Step S41 specifically involves:

[0069] First, the audio data is converted into a CQT image. Then, a target detection model based on deep learning is used to detect the first energy mutation position in the CQT image. The position of the energy mutation position in the CQT image is mapped to a time point in the audio data, thereby completing the detection of the first starting time point.

[0070] While this deep learning model can perform relatively well in detecting the start of singing, it can also produce some detection errors in practical applications. For example, when students begin singing, some may hum along with the accompaniment to better get into the singing state. In this case, the deep learning model may interpret the humming as singing, leading to a detection error because the actual start of singing occurs after the student begins humming. To address this issue, this invention combines a deep learning model with a traditional algorithm model.

[0071] Step S42 specifically involves:

[0072] Detect the first non-silent time point:

[0073] The first non-silent time point is detected using a traditional algorithm model. This involves using a Python dependency library to segment the audio into segments based on the silence points in the audio time domain. Then, we perform difference calculations on each segment and process the results, removing values ​​less than 20480. The first position of the remaining audio segment is the first non-silent time point to be detected.

[0074] Detection of time points where audio spectrum energy changes:

[0075] The system uses Python libraries to detect peak values ​​in the spectral energy of audio data. Each peak represents a time point. A difference algorithm is then used to calculate the intervals of the time series. Based on the number of intervals, if the number is less than 10, the system outputs the first time point and the maximum time interval of the sequence. If the number is greater than 10, the system takes the first 12 values ​​of the sequence, outputs the maximum time interval and the corresponding time point. Finally, based on the time interval, if the maximum time interval is greater than 2.96, the time point detected by the spectral energy abrupt change is used as the final time point detected by the traditional algorithm model; otherwise, the first non-silent time point of the audio is used.

[0076] The difference algorithm is a classic optimization algorithm that uses the next value in a sequence to subtract the previous value. This invention is used to detect the start time of an audio recording, and it is used to find the maximum interval value in the sequence, thereby finding the true start time of the audio recording.

[0077] Step S50 specifically includes:

[0078] Step S51: Using a Python dependency library, slice the audio data according to the silence points in the audio time domain and convert it into a spectrum image. Record the size and length of the spectrum image for later time point mapping.

[0079] Step S52: After performing second harmonic processing on each of the spectrum images, the abrupt change positions and decay positions of energy in the spectrum images are detected by the target detection model. The abrupt change position is taken as the starting position of the note, and the position with a value of 0 after the decay position is taken as the ending position of the note. That is, the position corresponding to the decay from the decay position to nothing is the ending position. Since the resolution of the spectrum images is low, second harmonic processing is performed on the spectrum images.

[0080] Step S53: Convert the start position and end position into corresponding note timing points; that is, stitch all the slices together, mark all the start and end positions using a linear proportional calculation method, and then calculate the note timing points of the corresponding positions based on the marked positions.

[0081] Step S54: Detect the peak time point of the spectral energy of the audio data using the Python dependency library, and correct and supplement the note time points based on the peak time point of the spectral energy.

[0082] Correcting the starting position of notes: Using Python dependency libraries to detect the peak time points of the spectral energy of audio data, based on the starting and ending positions of each note, check if there is a peak time point of the spectral energy falling in the middle of the starting and ending positions. If so, select the first peak time point of the spectral energy, then calculate the frequency sequence between the starting position of the note and the peak time point of the spectral energy, and then select the first non-empty value position as the final starting position (note time point) of the note.

[0083] Correcting the ending position of notes: Using Python dependency libraries to detect the peak time points of the spectral energy of audio data, based on the ending position of each note and the starting position of its next note, check if there is a peak time point of spectral energy falling within it. If so, select the first peak time point of spectral energy, then calculate the frequency sequence between the ending position of the note and the peak time point of spectral energy, and select the last non-empty value position as the final ending position (note time point) of the note.

[0084] To supplement missed notes: Use Python dependency libraries to detect the peak time points of the spectral energy in the audio data. Check if any peak time points fall at the end position of each note and the beginning position of the next note. If the number of peak time points is greater than 1, then from the second peak time point onwards, every two peak time points form the start and end time points of a note, and then these are added to eliminate missed notes.

[0085] Output the final note timings: Add the corrected and supplemented note timings, and then output all note timing information to the subsequent programs.

[0086] Step S60 specifically involves:

[0087] Based on the time points of each note, the audio data is sliced ​​to obtain several second audio segments. The third pitch value sequence of the second audio segments is extracted using a Python dependency library. The mode of each third pitch value sequence is used as the audio feature of the audio data.

[0088] The sheet music data is scanned using a sheet music recognition tool to identify the pitch value and time point corresponding to each character in the sheet music data, thereby obtaining a sheet music file in JSON format.

[0089] Step S70 specifically involves:

[0090] The audio features are subjected to pitch dilation based on the sampling rate of the audio format (e.g., 44100 sampling rate for WAV format audio). Then, the pitch values ​​of the audio features and the score file are aligned using the Dynamic Time Warping (DTW) algorithm. Next, the pitch difference between the audio features and the score file is calculated. Based on the pitch difference, the pitch accuracy of the audio data is automatically scored, and the scoring results are generated and displayed.

[0091] Pitch value dilation is the process of stretching a pitch value according to the length of time at a given point in time to facilitate subsequent alignment work.

[0092] Dynamic Time Warping (VTW) is a widely used algorithm for solving two-sequence matching problems. It's a classic dynamic programming algorithm capable of quickly determining the correspondence between two sequences. This invention uses this algorithm to align the pitch sequence of a student's singing with the pitch sequence carried in a standard sheet music file, and then scores the pitch accuracy based on this alignment. This invention calculates the shortest similarity path between the two sequences using VTW, and then reselects the pitch sequence in the sheet music file based on the shortest path to obtain the aligned pitch sequences.

[0093] The scoring rules for automatically scoring the pitch accuracy of audio data based on the pitch difference are as follows:

[0094] If the pitch difference is between 0 and 0.5, the audio segment receives full marks; if the pitch difference is between 0.5 and 5, the audio segment receives dynamic marks, calculated as 0.5 * (1 - pitch difference / 5) * 100; if the pitch difference is greater than 5, the audio segment receives 0 marks.

[0095] The audio segments with scores are marked, and the scores of these audio segments are combined to generate the final score result.

[0096] In summary, the advantages of this invention are:

[0097] By verifying the presence or absence of singing in audio data, scoring audio data without singing is avoided, reducing the waste of computing resources. Noise reduction is performed on the audio data using a deep learning model, facilitating subsequent audio feature extraction. Non-singing segments are removed from the audio data to avoid scoring noise. The accuracy of note timing is effectively improved by detecting the note timing points in the audio data and correcting and supplementing them based on the peak time points of the spectral energy. Audio features are extracted from the note timing points, and the corresponding sheet music data is processed to obtain sheet music files. Finally, the audio data is automatically scored based on the sheet music files and audio features, generating and displaying the scoring results. This process involves quantifying audio features at a fine-grained note level, aligning the audio features with the sheet music files based on pitch values, calculating pitch differences, and then automatically scoring based on these differences. The scoring process combines machine learning, deep learning, traditional algorithms, relevant audio processing technologies, and audio feature extraction methods, ultimately greatly improving the accuracy of singing pitch scoring.

[0098] While specific embodiments of the present invention have been described above, those skilled in the art should understand that the specific embodiments described are merely illustrative and not intended to limit the scope of the present invention. Equivalent modifications and variations made by those skilled in the art in accordance with the spirit of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A multi-stage automatic pitch scoring method based on segmentation, characterized in that: Includes the following steps: Step S10: Obtain the audio data and sheet music data to be scored; Step S20: Verify whether there is singing in the audio data; Step S30: Perform noise reduction processing on the audio data using a deep learning model; Step S40: Remove the non-singing segments from the audio data; specifically including: Step S41: Detect the first starting time point of the audio data using a deep learning model; Step S42: Detect the second starting time point of the audio data using a traditional algorithm model; Step S43: Based on the first and second starting times, cut a first audio segment from the audio data, extract the second frequency of the first audio segment using a Python dependency library, convert the extracted second frequency into a second pitch value sequence, measure the length 'a' of the second pitch value sequence, measure the length 'b' of the second pitch value sequence after removing null values, and determine whether 'b / a' is greater than 0.

65. If so, use the smaller value between the first and second starting times as the third starting time; otherwise, use the larger value between the first and second starting times as the third starting time. Step S44: Remove the non-singing segments from the audio data before the third starting time point; Step S50: Detect the note timing of each note in the audio data; Step S60: Extract audio features from the audio data based on the time points of each note, and organize the score data to obtain a score file; Step S70: Automatically score the audio data based on the sheet music file and audio features, and generate and display the scoring results.

2. The multi-stage automatic pitch scoring method based on fragmentation as described in claim 1, characterized in that: Step S10 specifically involves: Obtain the audio data to be scored and the corresponding sheet music data.

3. The multi-stage automatic pitch scoring method based on fragmentation as described in claim 1, characterized in that: Step S20 specifically involves: The first frequency of the audio data is extracted using a Python dependency library. The extracted first frequency is converted into a first pitch value sequence. The length A of the first pitch value sequence and the length B of the first pitch value sequence after removing null values ​​are measured. It is determined whether B / A is greater than 0.

21. If yes, it means that the audio data contains singing, and the process proceeds to step S30. If no, it means that the audio data does not contain singing, a score of 0 is generated and displayed, and the process ends.

4. The multi-stage automatic pitch scoring method based on fragmentation as described in claim 1, characterized in that: Step S30 specifically involves: After the audio data is denoised once using the Demucus model, it is then denoised a second time using the Spleeter model.

5. The multi-stage automatic pitch scoring method based on fragmentation as described in claim 1, characterized in that: Step S50 specifically includes: Step S51: Using Python dependency libraries, slice the audio data according to the silence points in the audio time domain and convert it into a spectrum image; Step S52: After performing second harmonic processing on each of the spectrum images, the abrupt change position and decay position of energy in the spectrum image are detected by the target detection model. The abrupt change position is taken as the starting position of the note, and the position with a value of 0 after the decay position is taken as the ending position of the note. Step S53: Convert the start position and end position into the corresponding note time points; Step S54: Detect the peak time point of the spectral energy of the audio data using the Python dependency library, and correct and supplement the note time points based on the peak time point of the spectral energy.

6. The multi-stage automatic pitch scoring method based on fragmentation as described in claim 1, characterized in that: Step S60 specifically involves: Based on the time points of each note, the audio data is sliced ​​to obtain several second audio segments. The third pitch value sequence of the second audio segments is extracted using a Python dependency library. The mode of each third pitch value sequence is used as the audio feature of the audio data. The sheet music data is scanned using a sheet music recognition tool to identify the pitch value and time point corresponding to each character in the sheet music data, thereby obtaining a sheet music file in JSON format.

7. The multi-stage automatic pitch scoring method based on fragmentation as described in claim 1, characterized in that: Step S70 specifically involves: The audio features are subjected to pitch dilation based on the sampling rate of the audio format. Then, the pitch values ​​of the audio features and the score file are aligned using a dynamic time warping algorithm. Next, the pitch difference between the audio features and the score file is calculated. Based on the pitch difference, the pitch accuracy of the audio data is automatically scored, and the scoring results are generated and displayed.

Images